Longitudinal dermoscopic study employing smartphone-based image registration

ABSTRACT

The evolution of a skin condition over time can be useful in its assessment. In an illustrative arrangement, a user captures skin images at different times, using a smartphone. The images are co-registered, color-corrected, and presented to the user (or a clinician) for review, e.g., in a temporal sequence, or as one image presented as a ghosted overlay atop another. Image registration can employ nevi, hair follicles, wrinkles, pores, and pigmented regions as keypoints. With some imaging spectra, keypoints from below the outermost layer of skin can be used. Hair may be removed for image registration, and restored for image review. Transformations in addition to rotation and affine transforms can be employed. Diagnostic correlations with reference image sequences can be made, employing machine learning in some instances. A great variety of other features and arrangements are also detailed.

RELATED APPLICATION DATA

This application is a division of application Ser. No. 14/276,578, filedMay 13, 2014, which is a continuation of international applicationPCT/US14/34706, filed Apr. 18, 2014, which claims priority toprovisional application 61/978,632, filed Apr. 11, 2014. ApplicationSer. No. 14/276,578 is also a continuation-in-part of application Ser.No. 14/206,109, filed Mar. 12, 2014, which claims priority toprovisional applications 61/813,295, filed Apr. 18, 2013; 61/832,715,filed Jun. 7, 2013; 61/836,560, filed Jun. 18, 2013; and 61/872,494,filed Aug. 30, 2013. These applications are incorporated herein byreference.

INTRODUCTION

Medical diagnosis is an uncertain art, which depends largely on theskill and experience of the practitioner. For example, dermatologicaldiagnosis tends to be based on very casual techniques, like observationby doctor, or on very invasive techniques, like biopsies. Skin conditiondegrades with age. It is difficult for people to differentiate theeffects of normal aging from disease. This leads to lots of worry andunnecessary doctor visits. More rigorous diagnostic techniques can beapplied to educate the public, assist medical professionals, and lowerhealth care costs.

An example is diagnosis of diseases evidenced by skin rashes and otherdermatological symptoms. A skilled dermatologist may be able toaccurately identify dozens of obscure conditions by their appearance,whereas a general practitioner may find even some common rashes to beconfounding. But highly skilled practitioners are sometimes puzzled,e.g., when a rash appears on a traveler recently returned from thetropics, and the practitioner has no experience with tropical medicine.

Some of the dimensions of differential diagnosis in dermatology includelocation on body, color, texture, shape, and distribution. Otherrelevant factors include age, race, sex, family tree, and geography ofperson; and environmental factors including diet, medications, exposureto sun, and occupation. Many skin conditions have topologies andgeographies that can be mapped in various dimensions, including depth,color and texture.

The prior art includes smartphone apps that are said to be useful indiagnosing skin cancer. Some rely on computerized image analysis. Othersrefer smartphone snapshots to a nurse or physician for review. Theformer have been found to perform very poorly. See, e.g., Wolf et al,Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection,JAMA Dermatology, Vol. 149, No. 4, April 2013 (attached to application61/872,494).

In accordance with one embodiment of the present technology, imagery ofdermatological conditions and other enrollment information is compiledin a crowd-sourced database, together with associated diagnosisinformation. This reference information may be contributed by physiciansand other medical personnel, but can also be provided by the lay public(e.g., relaying a diagnosis provided by a doctor).

A user submits a query image to the system (typically with anonymousenrollment/contextual information, such as age, gender, location, andpossibly medical history, etc.). Image-based derivatives are determined(e.g., color histograms, FFT-based metrics, etc.) for the query image,and are compared against similar derivatives for the reference imagery.In one arrangement, those reference images whose derivatives mostclosely correspond to the query image are determined, and theirassociated diagnoses are identified. This information is presented tothe user in a ranked listing of possible pathologies. In someembodiments, when the user's submitted query image and associatedinformation is analyzed by the system and several likely diagnosesidentified, the system may provide specific questions (guided by theresults of the current analysis) to the user, or requests for additionalimages, to help distinguish among the candidate diagnoses.

In a variant arrangement, the analysis identifies diseases that are notconsistent with the query image and associated information. Again, thisinformation is reported to the user (expressly; not simply by omission)and may include a risk profile that conveys statistical and/orqualitative measures of risk about such information.

In some embodiments, the imagery is supplemented with 3D informationabout the surface topology of the skin, and this information is used inthe matching process. Such 3D information can be derived from theimagery, or may be separately sensed.

Depending on the specificity of the data, and the size of thecrowd-sourced database, 90%, 98%, or more of candidate conditions can beeffectively ruled-out through such methods. A professional using suchtechnology may thus be able to spare a patient expensive and painfultesting (e.g., biopsies), because the tested-for conditions can bereliably screened by reference to the large corpus of reference imageryand associated knowledge generated by the system. Similarly, a worrieduser may be relieved to quickly learn, for example, that an emergingpattern of small lesions on a forearm is probably not caused byshingles, bedbugs, malaria or AIDs.

In some embodiments, the knowledge base includes profile informationabout the subjects whose skin conditions are depicted. This profileinformation can include, e.g., drugs they are taking, places they havevisited in the days leading up to onset of symptoms, medical history,lifestyle habits, etc. When a user submits a query image, and the systemidentifies reference imagery having matching derivatives, the system canalso report statistically-significant co-occurrence information derivedfrom the profile information. For example, the system may report that27% of people having a skin condition like that depicted in the user'squery image report taking vitamin A supplements.

In some embodiments, the co-occurrence information is broken down bycandidate diagnoses. For example, the system may report that the topcandidate diagnosis is miliaria X (42% chance). 35% of people with thisdiagnosis report having been in the tropics in the 30 days prior toonset of symptoms, and 25% report occasional use of hot tubs or saunas.The next top candidate diagnosis is tinea Y (28% chance). 60% of peoplewith this diagnosis report having chicken pox as a child. Suchco-occurrence information can help in making a differential diagnosisfrom among the offered alternatives.

Sometimes a patient is less concerned with the diagnosis than simplywanting to be rid of an affliction. Thus, some embodiments of thetechnology do not attempt to identify, or rule-out, particulardiagnoses. Instead, they simply seek to identify correlated factors fromthe knowledge base created from information from users, image analysis,and crowd-sourced data, so that possibly causative factors might beaddressed (e.g., by suspending intake of supplemental vitamin A, in theexample given above).

Typically, the user-submitted information is added to the knowledgebase, and forms part of the reference information against which futuresubmissions are analyzed.

While described primarily in the context of skin imagery, the principlesof the present technology are likewise applicable with anyphysiologically-derived signals. An example is audio. Audio signalsinclude heart sounds and other cardiovascular sounds (including murmurs,bruits, and other blood flow noises), lung and other respiratory sounds(including crackles, rales, rhonchi, wheezes, coughs, snoring and otherair flow noises), bowel and digestive sounds, joint noises (e.g., popsand creaks), as well as speech and other vocalizations.

The foregoing and other features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates components of one implementation of the technology,including plural remote terminals (e.g., smartphones), and one or morecentral systems.

FIG. 2 illustrates conceptual organization of an exemplary diagnosticsystem using technology disclosed herein.

FIG. 3A shows a banknote, and FIG. 3B shows an excerpt from thebanknote.

FIG. 4 shows normalized reflectance plots for the FIG. 3B banknoteexcerpt, and for a white envelope.

FIG. 5 is a schematic sectional view of a full-body imaging booth.

FIGS. 6A and 6B are views depicting features of alternate imagingbooths.

FIGS. 7, 8 and 9 detail sequences of smartphone screen displays thatprovide illumination of different spectral characteristics, fromdifferent positions relative to the smartphone camera.

FIG. 10 details how light from different parts of a smartphone screendisplay illuminates a feature on a skin from different angles.

DETAILED DESCRIPTION

FIG. 1 shows a hardware overview of one embodiment employing principlesof the present technology. Included are one or more user terminals(e.g., smartphones), and a central system.

As is familiar, each smartphone includes various functionalmodules—shown in rectangles. These include one or more processors, amemory, a camera, and a flash. These latter two elements are controlledby the processor in accordance with operating system software andapplication software stored in the memory.

The central system similarly includes one or more processors, a memory,and other conventional components. Particularly shown in FIG. 1 is aknowledge base—a database data structure that facilitates storage andretrieval of data used in the present methods.

One aspect of the present technology includes the central systemreceiving first imagery depicting a part of a human body that evidencesa symptom of a pathological condition (e.g., skin rash or bumps). Thisimagery (and its image metadata) can be uploaded to the central systemfrom one of the user terminals using commonly available image submissionmeans and enrollment. The image then is processed to derive one or moreimage parameter(s). A data structure containing reference information isthen searched, for reference image data that is parametrically similarto the first imagery. Based on results of this search, one or moreparticular pathological conditions that are not the pathologicalcondition evidenced by the depicted part of the human body areidentified. Resulting information is then communicated to theoriginating user terminal.

In some cases, it may be useful to present “similar” images to the user,along with any accompanying diagnoses. In these cases, it is importantto ensure that the presented images are relevant to the user. In manyimplementations, a machine learning approach will be suitable fordetermining candidate diagnoses. Many features, whether part of the rawuser images, or processed versions thereof, are presented to the machinelearning algorithm. Additional features, representing age, gender, race,height, weight, etc., are also likely to be input to the algorithm. Themachine learning algorithm can output a set of candidate diagnoses whichbest match the user's images and additional information. Ifrepresentative images are to be presented to the user as representativeof their diagnoses, the images should be chosen from the database notjust based upon visual similarity, but also based upon how well thedatabase image's associated additional features (age, gender, etc.)match the user.

(For expository convenience, the terms image, imagery, image data, andsimilar words/expressions, are used to encompass traditional spatialluminance/chrominance representations of a scene, and also to encompassother information optically captured from a subject. This can include,for instance, 3D microtopology. Such terms also encompass suchinformation represented in non-spatial domains, e.g., FFT data, whichrepresents the information is a spectral domain.)

The derived image parameter(s) can be of various types, with some typesbeing more discriminative for some pathologies, while other types aremore discriminative for others.

One sample derived image parameter is a color histogram. This histogrammay be normalized by reference to a “normal” skin color, e.g., assampled from a periphery of the area exhibiting the symptom.

Particular types of suitable color histograms are detailed in Digimarc'sU.S. Pat. No. 8,004,576. One such histogram is a 3D histogram, in whichthe first and second histogram dimensions are quantized hues (e.g.,red-green, and blue-yellow), and the third histogram dimension is aquantized second derivative of luminance.

Desirably, the imagery is spectrally accurate, so that hue-based imagederivatives are diagnostically useful. One low cost approach toacquiring such imagery is by gathering multiple frames of imagery underdifferent, spectrally tuned illumination conditions, and processingsame, as detailed in co-pending application Ser. No. 13/840,451, filedMar. 15, 2013 (now published as 20130308045), and Ser. No. 14/201,852,filed Mar. 8, 2014.

Another type of derived image parameter is a transformation of theimagery into a spatial frequency domain representation (e.g., FFT data).Such representation decomposes the image into components of differentfrequencies, angular orientations, phases and magnitudes (depending onthe manner of representation). Parameters of this type are particularlyuseful in discerning skin textures, which are often useful as diagnosticcriteria. The decomposition of the image into such spatial frequencycomponents can be conducted separately in different channels, e.g.,yielding two-, three- or more-binned representations of different imagechrominance and luminance planes. (More than the usual tri-color imagerepresentations can be used. For example, the image may be representedwith 4-20 different color channels.)

Still another image derivative is wavelet transform data. Suchinformation is again a decomposition of the image information into acollection of orthonormal basis functions—in this case wavelets.

A variety of other domain transformations can similarly be applied tothe imagery to serve as the basis of image metrics for matching.

Yet another image derivative is blob analysis. One form of such analysisinvolves “region growing.” A particular method, practiced in the pixeldomain, involves selecting a seed pixel, and adding to a blob all of thecontiguous pixels whose values are within a threshold value range of theseed pixel, e.g., plus or minus three digital numbers in luminance, on a0-255 scale. This process can be repeated for seed pixels throughout theimage. The seed pixels can be selected based on color or other parameter(e.g., a local maxima in image redness or contrast), or may be chosenrandomly. What results is a pattern of 2D regions whose shape and scaleparameters are useful as diagnostic indicia.

A particular image metric derived from blob analysis is a histogramidentifying frequency of occurrence of different shapes. Shapes may beclassified in various fashions. A simple two-class division, forexample, may distinguish shapes that have exclusively convex boundaries(e.g., circles and ovoids) from shapes that have a concave aspect topart of their peripheries (e.g., blobs that have one or moreinwardly-directed dimples). Much more sophisticated techniques arecommonly used in blob analysis; an example is a histogram of orientedgradients. (See, e.g., Dalal, et al, Histograms of Oriented Gradientsfor Human Detection, IEEE Conference on Computer Vision and PatternRecognition, pp. 886-893, 2005.)

(Commonly, such blob analysis is performed using a support vectormachine method, which classifies shapes based on a set of referencetraining data.)

While luminance was used in the foregoing example, the technique canalso be practiced in a particular color channel, or in Boolean logicalcombinations of color channels (e.g., add to the blob region thosepixels whose value in a 500 nm spectral band is within 3 digital numbersof the seed value, OR whose value in a 530 nm spectral band is within 5digital numbers of the seed value).

Similar methods can be practiced in other domains, such as using arepresentation of imagery in the spatial frequency domain.

All such image derivatives (metrics) can be computed on differentscales. One scale is across the totality of an image. Another is todivide the image into hundreds of portions, and compute the metrics foreach such portion. The same image can be re-divided into tens ofthousands of portions, with the metrics again recomputed. These portionsmay be of any shape; rectangular is often computationally efficient, butothers can be used. The portions may be disjoint, tiled, or overlap. Ifcomputational constraints require, the finer scale metrics can becomputed on a subset of all such regions, such as on a random selectionof 1% of 100,000 regions.

As indicated, the image derivatives can be computed on different colorchannels. Using methods detailed in the pending patent application Ser.No. 13/840,451 (now published as 20130308045) and Ser. No. 14/201,852,for example, an image can be captured and accurately decomposed intofive or ten or more different spectral bands—each of which may havediagnostic utility. Such spectral-based analysis is not limited to thevisible spectrum; infrared and ultraviolet data is also useful.

(Ultraviolet light is absorbed by melanin. Thus, illumination with UVcan reveal irregular pigment distribution, which can aid, e.g., indefining the borders of melanoma.)

(The CMOS and CCD sensors used in conventional digital cameras aretypically responsive well into the infrared, provided there is no IRfiltering.)

The image, and image derivatives, can also be based on polarized lightphotography.

Bag-of-features techniques can be applied to the image derivatives,e.g., as detailed in Csurka, et al, Visual Categorization with Bags ofKeypoints, ECCV, Workshop on Statistical Learning in Computer Vision,2004.

Another image derivative is feature size. Dimensions (e.g., diameter) oflesions and other visually-distinguishable skin features can be assessedfrom imagery, and this data included with the derivative image data.(The diagnostic profile of a feature is often dependent on its size.)

FIG. 2 is an excerpt of a conceptual view of a reference database. Itincludes a variety of records (rows), each comprising a set of datarelating to a reference subject.

The first column contains an image (or a set of images) depicting adermatological condition of the subject. An image can comprise, e.g., a10 megabyte color TIF file.

The second column shows some of the image derivatives computed from theimage. The naming convention gives semantic information about the typeof data, e.g., indicating whether it is histogram or FFT data, and thecoordinate of a tiled sub-region of the image from which the data wasderived.

The third column shows the location on the subject's body from which theimage was captured.

The fourth column shows, if available, a diagnosis of the referencesubject's affliction. For some entries, no diagnosis is provided.

The fifth column shows additional user metadata. Examples includedemographic information (e.g., age, gender, weight, height, race,residence location by zip code), and other profile data about thesubject. This can include drugs taken in the past thirty days, anyon-going medical conditions, foods introduced into the subject's diet inthe past thirty days, travel within the past sixty days, lifestyleactivities, environmental exposures, family medical history, etc.

It will be seen that information in the fourth and fifth columns istagged using XML-style descriptors, to provide for extensibility and tofacilitate text parsing.

A query image submitted by the user can similarly be accompanied by thebody location and other user metadata information shown in FIG. 2.

(Not shown in FIG. 2, but typically present in the knowledge base foreach image, is metadata concerning the image capture parameters, e.g.,in the standard EXIF format.)

In an illustrative embodiment, a server system determines similarityscores between a query image and each of many reference images. Onecomponent of such a score can be based on the reciprocal of a Euclideandistance between an image derivative from the query image and acorresponding image derivative for a reference image, in the imagederivative feature space. Since each image may have thousands ofderivatives (e.g., based on different regions and color channels), therecan be many thousands of such components (e.g., comparing a histogram ofregion 1 of the query image with histograms of regions 1-1,000 of areference image, and likewise for region 2 of the query image, etc.).Typically, such feature similarity metrics that fall below astatistically significant threshold are ignored.

In computing a similarity score for a reference image (i.e., relative toa query image), some image derivatives are weighted more heavily thanothers. For example, the weight given to a particular correspondencebetween a pair of image derivatives can depend on the scale of theportions between which similarity was found. The larger the feature, themore weight is typically given (e.g., in linear or exponentialproportion to feature size). Similarly, some indicia are morediagnostically relevant than others. Spectral data at 500 nm may be morediscriminative than spectral data at 700 nm, and may be given acommensurately greater weight, etc. Weightings can be calculatedrecursively, accounting for feedback from users of the system aboutcorrelations.

A sampling, or all, of the reference images in the database are thusscored relative to the query image. In an illustrative embodiment, thereference images that are scored in the top 5%, or 0.5%, of the universeof evaluated reference images are thereby identified. Associated usermetadata for this set of reference images is then analyzed.

Naturally, many of the top matches will be errant. Some of theinformation in the database may also be incorrect. In an informationtheoretic sense, the data will be very noisy. But in the aggregate, overa large data set, statistically significant and useful correlations willbe evident.

For example, analysis of the top-scoring set of reference images mayfind that 40% are associated with diagnostic tags indicating that theydepict the condition known as tinea versicolor, and 23% may be similarlytagged as depicting pityriasis rosea. 25% of the top-scoring referenceimages may be associated with diagnostic tags indicating that thereference subject was taking the blood pressure medicine Atenolol.

Of course, a person either has a condition, or doesn't. A person doesn'tsuffer from “40% tinea versicolor, 23% pityriasis rosea, etc.” But sucha ranked presentation of candidates provides specific hypotheses thatcan then be further investigated.

A statistical breakdown of such correlations is typically provided tothe user—in one or more rank-ordered sets. For example, the user may bepresented with a rank-ordered listing of the top five or ten possiblediagnoses—each including a stated probability based on frequency ofoccurrence from the top-matching reference image set. Similar listingsmay be presented for demographic information and other profile data(e.g., drug correlations, diet correlations, lifestyle correlations,etc.).

It will be understood that many skin conditions are themselves symptomsof non-skin disorders. A familiar example is skin jaundice, which may beassociated with liver failure. Such non-skin diagnoses should also bereflected in the knowledge base, and in the results reported to theuser.

The absence of apparent correlation can additionally, or alternatively,be reported to the user. If less than 0.03% of the reference images inthe top-scoring set are associated with tinea versicolor, whereas thiscondition has a much greater frequency of occurrence in the fullreference image set (e.g., 1.5%), then the user can be informed that theskin condition is most likely not tinea versicolor. Likewise with drugs,diet, lifestyle, etc. (The particular threshold used in such evaluationcan be determined empirically.)

The information presented to the user can also include samples ofclosely-matching reference imagery—and the diagnosis (if any) associatedwith each.

Another method makes use of changes in the user's depicted symptoms overtime. In such a method, the user submits two images to the system—aninitial one, and a second one taken at a later time. The systemdetermines data about a change in the depicted skin symptom betweenthese two times based on the submitted imagery. This determined data isthen used in further refining diagnostic information.

Thus, for example, if purpura on the skin enlarge in size during thecourse of a disease, this is evidence in favor of certain candidatediagnoses, and contrary to other candidate diagnoses. Three or moretime-based images can likewise be used.

Crowd-sourced data gathering, with subsequent scoring and statisticalcalculations as described herein, should both be initiated as well asmanaged as it evolves. Expert medical practitioners have the opportunityto “seed” such databases with known imagery examples of a variety ofafflictions, paying a great deal of attention to ensuring a wide rangeof angles, lighting conditions, parts of the body, camera models, etc.This can involve the submission of hundreds, thousands or even moreimages with clinically derived examples of the major and less majorcategories of affliction. Likewise, even as the crowd sourced imagerygrows with time and may soon wind up dwarfing any original “groundtruth” imagery, expert practitioners can still submit known examples ofafflictions into the up-to-date crowd-sourced service as described,witness the results of the returned information, then proceed to tune ormodify various weighting factors, scoring approaches, extensions to XMLfields, etc., thereby managing the diagnostic accuracy of the overallservice as more and more clients begin to use the service.

It will be recognized that certain embodiments of this technology differfrom earlier crowd-sourced dermatological efforts in various ways. Forexample, some of the earlier work compiled a crowd-sourced collection ofimages that were each accompanied by professional diagnosis data. Theillustrative embodiment has no such requirement. Similarly, otherearlier work employed a “crowd” to offer plural human assessments ofsubmitted images, from which a consensus conclusion was derived. Theillustrative embodiments do not require such plural human assessments.

Unprecedented knowledge will be revealed as the present system grows tolarge scale. Error tends towards zero as the universe of data growslarge.

Data Capture

The capturing of data from skin can employ known and forthcoming imagingtechnologies. A simple one is a smartphone camera. Accessory optics maybe employed to provide better close-up capabilities. Other digitalcameras—including those on headworn devices—can also be used.

Exemplary smartphones include the Apple iPhone 5; smartphones followingGoogle's Android specification (e.g., the Galaxy S5 phone, manufacturedby Samsung, and the Google Moto X phone, made by Motorola), and Windows8 mobile phones (e.g., the Nokia Lumia 1020, which features a 41megapixel camera).

Some embodiments employ modular mobile device technology, such asGoogle's Project Ara (which derives from the Phonebloks conceptdeveloped by Dave Hakkens; see, e.g., YouTube video oDAw7vW7H0c and thephonebloks<dot>com web site). In such arrangements, a mobile device iscomprised of detachable components, which can be added, changed orupgraded as needs dictate. Through such technologies, a device can beassembled that includes sensors of the sorts detailed in thisdisclosure, especially adapted for physiologic data capture.

Imagery employed in the present technology may be in JPEG format, butpreferably is in a higher quality form—such as RAW or TIF.

The smartphone or other user device can compute some or all of thederivative information from the sensed data before sending data to theremote database, or the central system can perform such calculations,based on provided sensor data. Or these tasks can be distributed—partperformed on one platform, and part on another.

In addition to smartphone cameras, image capture can employpurpose-built hardware. Examples are disclosed in patent publication20110301441. Commercial products include the Dermograph imager byMySkin, Inc., and the Handyscope by FotoFinder Systems. The latter is anaccessory for the Apple iPhone 5 device and includes built-inillumination—optionally cross-polarized. It is capable of capturing bothcontact images (with the device touching the skin), and non-contactimages. A variety of other dermatoscopy (aka epiluminescence microscopy)hardware systems are known.

In some arrangements, a physical fixture can be provided on the imagingdevice to help establish a consistent imaging distance to the skin. Arigid black, white or clear plastic cowl, for example, can extend fromthe camera lens (and optionally flash) at one end, to an opening that isplaced over the skin, for controlled-distance imaging.

Software on the smartphone may employ known auto-focus technology to setan initial image focus, and can warn the user if the camera is unable toachieve proper focus. However, some auto-focus algorithms are easilyfooled into focusing on dark hair that may rise above the skin surface.Accordingly, it is preferable to capture several still imageexposures—one at the nominal auto-focus setting, and others that arevaried under software control from that position, e.g., at focal planesplus and minus two and four millimeters from the auto-focus setting.Known computational photography techniques can combine such images toyield a composite image with an extended depth of field, as detailed,e.g., in Jacobs et al, Focal Stack Compositing for Depth of FieldControl, Stanford Computer Graphics Laboratory Technical Report 2012-1,attached to application 61/872,494. (Other extended depth of fieldtechnologies can also be employed, e.g., as detailed in U.S. Pat. Nos.7,218,448, 7,031,054 and 5,748,371.)

Similarly, the software can employ exposure-bracketing, since somefeatures may more easily be distinguished in exposures taken one or twof-stops above, or below, an autoexposure setting. Known high dynamicrange methods can be employed to composite such images into an enhancedimage frame.

In some arrangements, a camera's frame capture is triggered based onstability. A stability metric can be based on data from a smartphonesensor (e.g., an accelerometer). Or it can be based on analysis of theviewfinder image data. (The Apple iPhone device includes motionestimation hardware, which is most commonly employed for MPEG videocompression, but which also can track features in an image frame toassess image stability.)

While imagery captured by mobile cameras is a focus of this disclosure,it will be recognized that imagery captured by whole body scanningsystems can likewise be employed. Canfield Scientific is among thecommercial providers of whole body scanners.

In between smartphones and whole-body scanners are a range ofintermediate imaging systems. One is an automated apparatus that mightbe found in a doctor's office or pharmacy, which serves to captureimagery from a user and submit it to the central system for analysis, asdetailed herein. Such apparatus (which may be, e.g., a stand-alonekiosk, or integrated into a weight scale in a doctor's office—capturingfrontal face and neck imagery each time a patient is weighed) can bemore sophisticated than that found in most smartphones, e.g., providingcontrolled spectral illumination (e.g., as in application Ser. No.13/840,451 (now published as 20130308045) and Ser. No. 14/201,852),thermal imaging, etc. It may provide the user with a hardcopy printoutof the results. Such an apparatus may be available for free use, or anominal charge may be collected (e.g., by coin, dollar, or credit card).

As is familiar to artisans, various photosensitizers (e.g.,aminolevulinic acid) can be applied to the skin, to highlight certaintumors, etc., such as by changing their absorbance and fluorescencespectra.

In some methods, the user moves a smartphone over a body area, while thecamera captures imagery multiple frames of imagery. From the differentviewpoint perspectives, 3D information about the skin's surface relief(topology) is discerned, e.g., using familiar stereoscopy techniques.Google's patent publication 20130201301 details one such arrangement forcreating 3D imagery from smartphone images captured at differentviewpoints. Known Simultaneous Localization and Mapping (SLAM) andStructure from Motion (SFM) techniques can also be employed—revealingscale as well as shape. Such a 3D data representation can be virtuallyflattened, using cartographic techniques, for analysis and rendering tothe user.

Patent application Ser. No. 13/842,282, filed Mar. 15, 2013, details howthe sensor in a moving device can be mounted on a MEMS-actuatedpedestal, and moved in a cyclical fashion synchronized with the framecaptures, to counteract motion blur. The multiple frames of imagerycollected in such a capture arrangement can be combined to yield anenhanced resolution image (e.g., as is taught in Digimarc's publishedpatent application 20080036886 and in U.S. Pat. Nos. 6,570,613 and5,767,987).

Other 3D sensing arrangements are known, e.g., as identified incopending application Ser. No. 13/750,752, filed Jan. 25, 2013 (nowpublished as 20130223673).

Above-noted patent application 13/842,282 details a particularlyadvantageous 3D camera sensor, employing photosites that are spectrallytuned—typically providing spectral responses at many more differentwavelengths (e.g., at eight different wavelengths—some of which may beoutside the visible range) than typical tri-stimulus (red/green/bluecolor-filter array) sensors of the previous art.

Another approach to 3D sensing is via an instrument that is touched tothe skin, causing a membrane to deform in correspondence with the skinsurface texture, forming what may be termed a skin print. Publishedpatent application 20130033595 details such an arrangement, including acamera that captures imagery from the back side of the membrane, underoblique illumination that emphasizes the texture topography. See alsoJohnson, et al, Retrographic Sensing for the Measurement of SurfaceTexture and Shape, 2009 IEEE Conf. on Computer Vision and PatternRecognition (attached to application 61/872,494). Such apparatus is nowavailable from GelSight, Inc., of Cambridge, Mass., and may eventuallybe integrated into cell phones and other wearable computer systems.

Skin topology measured using such skin print techniques is believed tohave a higher sensitivity and specificity for machine-basedidentification of certain skin conditions, as compared with 2D colorimagery. Although “ground truth” skin topographies, which associateparticular topographies with particular expert physician evaluations,are not yet available, these are expected to be forthcoming, when theutility of such measurements becomes widely known. Thus, another aspectof the present technology includes aggregating skin prints for a varietyof medical conditions in a reference database—at least some of whichalso include expert diagnoses associated therewith. A related aspectinvolves deriving features from such reference prints, and then usingsuch features in judging statistical similarities between a query skinprint submitted by a user and the reference skin prints, to identifycandidate diagnoses and other correlated information—as describedearlier.

Skin surface minutiae can also be sensed otherwise, such as by systemsfor capturing human fingerprints. Examples are known from the publishedpatent applications of AuthenTec (subsequently acquired by Apple),including applications 20120085822 and 20110309482. Such sensors arealready included in many laptop computers, and will doubtless soonappear in smartphones and the like.

Another image data collection technique comprises a flexible sheet withorganic transistor circuits. The circuits can comprise photodetectors,as detailed, e.g., in Fuketa, et al, Large-Area and Flexible Sensorswith Organic Transistors, 5th IEEE Int'l Workshop on Advances in Sensorsand Interfaces, 2013, and Baeg et al, Organic LightDetectors-Photodiodes and Phototransistors Advanced Materials, Volume25, Issue 31, Aug. 21, 2013, pp. 4267-4295, (both attached toapplication 61/872,494), and in references cited therein. Such media canalso include integrated OLED photodetectors—providing controlledillumination.

As earlier noted, polarized light photography can also be useful withthe present technology. This can be implemented with polarizedillumination, or with one or more polarizers on the camera or imagesensor. (See, e.g., Gruev, et al, CCD Polarization Imaging Sensor withAluminum Nanowire Optical Filters, Optics Express 18.18 (2010):19087-19094, which details a sensor having a polarizing filterassociated with each pixel. The filters have four differentorientations, offset by 45 degrees. By reading data fromdifferently-filtered sets of pixels, different image polarizations aresensed.) By sensing imagery at different polarizations, different imagefeatures can be revealed and different image effects can be achieved(e.g., increased contrast). Some research also indicates that polarizedlight, when reflected, has two orthogonal components—one due to the skinsurface morphology, and the other “back-scattered” from within thetissue.

While a user of the detailed system can submit a single image foranalysis, it is sometimes preferable to submit several. As noted, thesemay comprise differently-focused, or differently-exposed images. Theycan also comprise lesion-centered images from different viewingdistances, e.g., a close-up (e.g., where the lesion spans 25% or more ofthe image width), a mid-view (e.g., where the lesion spans between 5 and25% of the image width), and a remote view (e.g., where the lesion spansless than 5% of the image width).

The remote view will typically show a sufficiently large body excerptthat the location of the lesion (e.g., arm, foot, hand, face) can bedetermined using known anatomical classification techniques. (Manysmartphone operating systems, including those from Apple, include facialrecognition capabilities—which begin by recognizing a face in an image.)Such lesion location data can then automatically be entered into theknowledge base, without requiring entry of such information by the user.(In other embodiments, software can present the user with a 3D avatar onwhich the user virtually draws, or taps, to indicate locations of skinlesions.) Seeing the lesion in the context of an identifiable body partalso provides context from which the size of the lesion can beestimated. E.g., the average man's palm is 3.05 inches across,permitting the size of a lesion depicted in the same frame to bededuced.

As cameras and sensors continue to evolve, all three such views may becaptured from a single camera position. For example, a telephoto lensmay progressively zoom-out to capture the three just-referenced views.Or a high resolution sensor may have sufficient resolution that theformer two views can be extracted from a remote view image frame. Thesoftware application may automatically obtain the threeimages—controlling the zoom or cropping a high resolution image asappropriate. (Desirably, each view is at least 1000 pixels in width.)

In some embodiments, the smartphone software offers guidance to the userin capturing the images, e.g., directing that the user move the cameraaway from the body until the software's body part classifier is able toidentify the body part in the third view. Other direction, e.g.,concerning lighting and focus, can also be provided.

It is also sometimes diagnostically useful to consider images fromdifferent parts of the body. If a lesion appears on a user's forearm, asecond image may be submitted depicting the user's other forearm, or askin patch that is not normally exposed to the sun—such as under theupper arm. Difference metrics can then be computed that compare the skinparameters around the lesion site with those from the other site. Thesedata, too, can be submitted to the knowledge base, where similaritieswith other reference data may become evident.

Additional sensors will soon be commonplace on personal devices. Alreadyappearing, for example, are smartphones equipped with multiplemicrophones. As further detailed below, in conjunction with a smartphonespeaker, such a device is tantamount to an ultrasonic imager. Such adevice can be pressed to the user's skin, and the skin them stimulatedby ultrasonic sounds emitted by the speaker (or by other transducer—suchas a piezo-electric actuator). The microphones—sensing reflection ofsuch acoustic waves from inside the body, to the different microphonelocations—provide information from which imagery can be constructed.Such ultrasonic imagery is more grist for the present mill.

Similarly, liquid lenses (e.g., marketed by Philips under the FluidFocusbrand) may soon appear on smartphones, and enable new camera close-upand topological sensing capabilities.

In contact-based imaging (i.e., with the imaging apparatus touching theskin), the body location from which the image is captured can beelectrically sensed using small amplitude electrical waveforms insertedin the body by a wearable computer device—such as the Google Glassdevice, or a wrist-worn device. Especially if different signals areintroduced into the body at two locations, their distinctivesuperposition at the sensing site can accurately pinpoint the locationof such site.

Ambient Light, Pose, Scale, etc.

Color is an important diagnostic feature in assessing dermatologicalconditions. However, skin color, as depicted in captured imagery,strongly depends on the “color” of the light that illuminates the skin.While dermatologists can control illumination conditions in theiroffices, most consumer image capture is performed under widely varyinglighting conditions. To optimize performance of the detailedtechnologies, this variability should be mitigated.

Digital cameras commonly perform automatic white balance (AWB)adjustment. Various techniques are used. One technique examines thepixels in an image, and identifies one that is the brightest. This pixelis assumed to correspond to a white or shiny feature in the image, i.e.,a feature that reflects all of the incident light, without absorbing anyparticular color. The component color values of this pixel are thenadjusted to make it truly white (e.g., adjusting an RGB representationto {255,255,255}), and all other pixels in the image are remapped bysimilar proportions. Another technique averages all of the pixels in theimage, and assumes the average should be a shade of grey (e.g., withequal red, green, and blue components—if represented in the RGB colorspace). A corresponding adjustment is made to all the image pixels, sothat the average is remapped to a true shade of grey.

The former technique is ill-suited for skin photography because there istypically no white or specular pixel in the image. The latter techniqueis ill-suited because its premise—that the average pixel value isgrey—is not true for skin images.

Professional portrait photographers sometimes position a calibrationcard at the edge of a family group, where it can be cropped-out beforeprinting. The card includes various reference colors, including whiteand other known tones. Before printing, digital adjustments are made tothe image to bring the depiction of colors on the calibration card totheir original hues—thereby also color-compensating the portraitsubject.

Thus one approach to the ambient light issue is for a user to captureimagery from a calibration card, and send this image to the centralsystem, accompanying the skin image(s). The system can thencolor-compensate the skin image(s), based on the depiction of colors inthe calibration card image.

However, such calibration cards are not readily available, and cannottypically be electronically distributed to users for printing, due tocolor variability among consumer printers.

Applicant has found that various other materials can suffice in lieu ofcalibration cards.

One is a white envelope. The “white” on color calibration cards is acolorimetrically true white, whereas there is a great deal ofvariability in what passes for white among the lay public. But applicanthas found that white postal mail envelopes tend to be consistent intheir color—especially at the red end of the spectrum that is importantfor skin photography (there is more item-to-item variability at theviolet end of the range). While not “true white” in a colorimetricsense, such envelopes are generally consistent enough to serve as acolor reference.

So one approach to ambient light issues in consumer skin photography isto direct the user to capture imagery from a white envelope, under thesame lighting conditions as the skin photograph(s). This image can besent to the central system, where the skin photograph can becolor-corrected based on the envelope photograph.

The entire envelope needn't be photographed—just a fraction will do. Inone method, a part of the envelope substrate is torn or cut off, andplaced on the skin, within the camera's field of view. But sucharrangement a single image capture can suffice. Meanwhile, at thecentral system, the illumination-corrected, reflected color spectra froman assortment of white postal envelopes are captured and averaged, andused as reference data against which images received from end users arecolor-corrected.

(It will be recognized that placing a piece of a white envelope in thefield of view of a skin photograph can allow automatic white balancecorrection of the image by the camera—if the camera is using the formerof the above-described two AWB techniques. However, the details of aparticular camera's AWB algorithm are not generally known. The centralservice may, however, investigate the AWB techniques used by popularsmartphone cameras. By examining the metadata that commonly is packagedwith smartphone imagery, e.g., in the form of EXIF header data in animage file, the central system can determine the type of camera withwhich a user image was captured. If the image was captured from one ofthe cameras using the former AWB technique, and automated image analysisfinds that the image includes an area of white next to skin tone, thesystem can infer that appropriate color correction has already beenapplied by the camera.)

Another commonly available color reference—for those so-inclined—isoxygenated blood. Blood exhibits a consistent color spectrum despiterace and other variable factors. If a drop of blood is thick enough tomask the underlying skin pigment, its color can be sensed and again usedto reveal color information about the illumination.

Color calibration can also be performed with banknotes. Banknotes aretypically printed with extremely high tolerances, and consistent inkcolors. Desirably, a banknote excerpt having colors near the skin tonerange is employed. While US currency is commonly regarded as green, infact the US $20 bill has areas of skin-like tones to the left and rightof the Jackson portrait. (The US $10 has areas of reddish tones.)

In accordance with this method, the user captures images of the skin,and of a US $20 banknote, under the same illumination conditions. Boththe skin image and the banknote image are then sent to the centralsystem. The central system again compares the spectrum found in thereceived banknote image with reference data, and determines a spectralcorrection function detailing variance between the received banknoteimage and reference data. The system then applies this correctionfunction to the received skin image, to effect color correction.

Since color correction is primarily needed for skin tones, areas of abanknote lacking such colors can be omitted in performing the spectrummeasurement and correction. The central system can virtually identifythe relevant areas of the banknote artwork by reference to imagefeatures—such as SURF or SIFT keypoints, or by other pattern-matchingtechniques. An area bounded by such points can be virtually “clipped”from the artwork, and used as the basis for comparison against asimilarly-clipped set of reference data. FIGS. 3A and 3B show thebanknote artwork, and a representative clipped region spanning most ofthe skin tone region. This area is defined by “corner” features in theoriginal artwork (e.g., the upper right corner of the letter E in “ . .. PUBLIC AND PRIVATE;” the lower left corner of the A in AMERICA; etc.),and omits artwork that can vary between banknotes, i.e., the serialnumber.

The reference data is acquired by a reflectance spectroscopy techniquethat involves masking the banknote with a flat black mask—revealing onlythe clipped region—and illuminating with a light source whose spectrumis measured or otherwise known. Reflected light is sensed by aspectrometer, yielding a set of data indicating intensity as a functionof wavelength. This measured data is then adjusted to compensate for theknown spectrum of the light source.

FIG. 4 shows such a reference spectrum measured for both the Jacksonportrait excerpt shown in FIG. 3B (the lower line), and for a samplewhite postal envelope.

The contemplated system may serve users in diverse countries. Desirably,suitable calibration objects are identified so that one or more isavailable in each of these countries. The central system can examine theincoming imagery, and compare against a catalog of calibration objectsto recognize which object is being used. Thus, a customer may choose touse a Mexican 100 peso note as a reference, and the central system willrecognize same and apply the corresponding correction function.

It will be recognized that the above-described procedures for effectingcorrection of colors due to ambient lighting variability also effectcorrection of colors due to camera sensor variability. That is, if onecamera tends to emphasize greens, and another camera tends to emphasizereds, the imagery from both will be normalized to a consistent standardusing the arrangements detailed above.

The procedure employing a printed object in the image frame with theskin (as opposed to a white object) also allows the system to assess thebrightness of the imaged scene. Cameras have limited dynamic range. If ascene is too brightly lit, the camera's component red, blue and greensensors can no longer sense variability between different parts of theimage. Instead, each outputs its full maximum signal (e.g., 255, in an8-bit sensor). Faithful color sensing is lost. Similarly with too littleillumination; differently-colored areas are again indistinguishable. Byimaging a known printed object, such as a banknote, such over- andunder-exposure can be sensed (by comparison of detail in the sensedimagery with detail in reference imagery), and the user can be promptedto change the illumination and submit a new image, if needed.

If a known printed object is used as a color reference object, theobject artwork also enables other information to be sleuthed, such asscale, provided the object is depicted in the same image frame as theskin condition. To illustrate, the distance between the centers ofJackson's eyes on the US $20 banknote is 9 mm. If such a banknote isphotographed next to a lesion, and the distance between Jackson's eyesspans 225 pixels, and the lesion spans 400 pixels, then the lesion isknown to have a width of 16 mm. Dimensions of other features in theimage can be similarly determined.

If the printed object lies in the same plane as the skin, then the poseof the camera relative to the skin can also be determined—based onapparent geometrical distortion of the object. That is, if the cameraaxis is not perpendicular to the skin, then perspective distortion willcause features depicted in some parts of the frame to be larger, orsmaller, than would be the case with a perpendicular pose. By referenceto the known aspect ratio of features on the printed object, andcomparison with their aspect ratio in the captured imagery, the anglefrom which the image was captured can be sleuthed, and a correctivecounter-distortion can be applied. (The camera's optic function can alsobe considered in the analysis, to account for the expected apparentdistortion of features displaced from the center of the image frame. Forexample, the circular seal of the US Federal Reserve System, on the leftside of a banknote, may be subtly distorted from round—even with aperpendicular camera pose—if the seal is not at the center of the image.Such distortion is expected, and the analysis takes such normalartifacts of perpendicular poses into account.)

In the case of banknotes, still finer pose determinations can be made,based on security features that have different appearances withdifferent viewing angles. Color-shifting inks, security threads withmicroscopic lenses, and kinegrams, are of this sort. The central systemcan collect reference information quantifying the appearance of thesefeatures at different viewing angles. When a user-submitted image isrecognized to have such a banknote security feature depicted, itsrendering in the image can be matched with the reference information todetermine the angle at which it is being viewed—from which the viewingangle of the skin lesion can then be determined. (In many suchmeasurements, the color of the security feature shifts with viewingangle. Thus, it is desirable to first perform color-correction on theuser-submitted imagery, before analyzing pose in this fashion.)

Another calibration token that can be placed on the skin for imagecapture is a coin. Again, a variety of different coins may be recognizedby the central system—and from their known attributes, scale and posedeterminations can be made—just as with the banknote arrangementdescribed above. Also, many coins exhibit the specular reflection usedby many cameras for automatic white balance.

Other commonly available items that can be placed in the image frame toserve as props for color correction and/or scale measurement include thewhite cord of Apple USB cables and earbuds, and the USB plug itself. Theuser's thumb (or other finger) can also be put into the imageframe—providing a scale reference and also skin tone information.

Another approach to dealing with ambient light variability is to employthe smartphone's front-facing camera.

Smartphones are commonly equipped with two cameras—one on the front,facing the user, and one on the rear. The latter is typically used forcapturing skin imagery. But the former can be used to capture image datafrom which ambient lighting can be assessed. The field of view of thefront-facing camera can include a variety of subjects—making itsautomatic white balance determination more trustworthy than therear-facing camera (whose field of view may be filled with skin).

In accordance with this aspect of the technology, an automatic whitebalance assessment is made using the front-facing camera, and resultinginformation is then used in AWB-processing of skin imagery captured bythe rear-facing camera.

Still another approach to dealing with ambient light variability is touse flash illumination. The light emitting diodes (LEDs) used for cameraflashes have relatively consistent spectra among instances of aparticular model (e.g., iPhone 5 cameras). Reference data about flashspectra for popular camera models can be compiled at the central system.Users are then instructed to capture the skin image in low ambient lightconditions, with the camera flash activated. When the central systemreceives such imagery, it examines the header data to determine thecamera model involved, and flash usage. The system then applies a colorcorrection that corresponds to the flash spectrum for that model ofcamera.

Low ambient light can sometimes be difficult to achieve. And adaptingtechnical methods to the user, rather than adapting user actions to thetechnology, is generally preferable. In accordance with another aspectof the technology, flash is used in conjunction with ambient lightingfor color correction.

In one such method, two images are taken in quick succession—oneincluding an LED flash, and one not. (Video mode can be used, butresolution is typically better in a still image capture mode.) Bothimages include the ambient light, but only one includes the flash.Subtracting the two images leaves a difference image that is illuminatedby the LED flash alone-mitigating the uncertainty due to unknown ambientlighting. (The images can be spatially registered prior to subtraction,using known registration techniques, to account for slight motionbetween frames.) Again, the resulting image can be adjusted tocompensate for the spectrum of the LED flash. Software on the userdevice can effect such image capture, flash control, and differencingoperation.

Still another technique for color compensation is by reference tomeasured norms of skin coloration. While skin comes in a variety ofcolors, these colors comprise a tiny fraction of the universe ofpossible colors. This is particularly true when skin color isrepresented in the CIELAB color space. This range is narrowed stillfurther if the user's race is known, e.g., entered via the userinterface of a smartphone app, or recalled from stored user profiledata.

(In smartphones equipped with front- and rear-facing cameras, the formercan be used to capture a picture of the user—since the user typicallyoperates the phone facing towards the screen. Known techniques canassess the user's race (and gender) from facial imagery—avoiding theneed for the user to enter this information. See, e.g., Lyons, et al,Automatic Classification of Single Facial Images, IEEE Trans. on PatternAnalysis and Machine Intelligence, Vol. 21, No. 12, 1999, pp. 1357-1362(attached to application 61/872,494), and references cited therein. Therace assessment can be performed by smartphone app software, so that theuser's facial image is not sent from the phone.)

Since the user will typically frame a captured image so that a skincondition of concern is at the center, a better indication of the user'snormal skin color may be obtained by sampling away from the center,e.g., at the edges. An average color, based on samples taken from avariety of peripheral image locations, can be computed. (Samples shouldbe checked to assure that a location does not correspond to clothing orother non-skin feature. Color consistency and/or segmentation techniquescan be used.) This baseline skin color can then be checked againststatistical color norms—for the user's race, if known. If this baselinecolor is outside of the statistical norm (e.g., within which 99%, or99.9% of the population falls), then an adjustment is made to thecaptured imagery to shift the image colors so that the average fallswithin the norm. (The shift can move the average skin tone to thenearest edge of the norm region—as defined in the CIELAB color space—orto the center of the norm region.) For more on norms of skin colors, andrelated information, see, e.g., Zeng, et al, Colour and Tolerance ofPreferred Skin Colours, Color and Imaging Conference, Society forImaging Science and Technology, 2010 (attached to application61/872,494), and references cited therein.

While reference was made to assessing the size of skin features byreference to another article (e.g., a coin) in the image frame, othertechniques can also be used.

One is by photogrammetry, using camera and image data. For example, ifthe image metadata indicates the camera autofocus (subject distance) wasset at 6 inches, and the camera is known to capture a field of view thatis four inches wide in that focal plane, then an image feature thatspans a tenth of the width of the frame has a width of 0.4 inches.(Instead of autofocus information, data from a smartphone's proximitydetector can alternatively be used. Such detectors primarily rely oncapacitive techniques and are presently of short range, e.g., 2 cm., butlonger range sensors are under development.)

An allied technique, also from the field of photogrammetry, is bundleadjustment. (See, for example, Triggs et al, Bundle Adjustment—A ModernSynthesis, Proceedings of the International Workshop on VisionAlgorithms. Springer-Verlag., pp. 298-372, 1999). In bundle adjustmentalgorithms, multiple images taken from different locations and/ordirections are exploited to jointly produce estimates of the opticalview parameters of the camera(s) and a 2D or 3D model of the scenesimaged. While bundle adjustment originated in the photogrammetrycommunity, it has found much recent use in the computer science field,where is a fundamental component of shape from motion algorithms.Partial knowledge of the characteristics of the camera(s) can be used toimprove the accuracy of the scene model. In the case of skin imagescaptured with a smartphone, the multiple images may be made by passingthe camera over the patch of skin. By exploiting the redundancy providedby images taken from different perspectives, it is possible to providescale to the images. Since smartphones typically also have IMUs and/orgyroscopes, it is possible to improve upon the performance of thesealgorithms by feeding in this sensor information as side information.

Another scaling technique relies on known biometric norms. For example,in adults, the inter pupillary distance (the distance from the center ofone eye pupil to the center of the other) is about 62 mm. A variety ofother consistent biometric measurements are known (going back to thecarpenter's “Rule of Thumb” of antiquity), or can be gathered fromanalysis of data. Some are absolute measures (e.g., the inter pupillarydistance is about 62 mm), and others are ratios (e.g., the ratio offorearm length, to forearm plus hand length, is about 0.58). Some suchmeasures are tightly clustered, based on the user's gender and height.Image classification techniques can be applied to user imagery torecognize pupils, a thumb, a fingernail, a forearm, a hand, etc. Fromknown biometric measures, the size of a skin lesion can be inferred.

Other scaling techniques rely on such biometric norms, in conjunctionwith imagery from front- and rear-facing cameras. Consider a user takinga picture of a lesion on their forearm. The forearm can be recognizedfrom imagery captured by the smartphone camera. The smartphone ispositioned somewhere between the user's face and forearm, but itsdistance from the arm is unknown (disregarding auto-focus and otherestimation techniques). However, previous experimentation shows that atypical user tends to hold their smartphone camera about 12 inches fromtheir face, when viewing their forearm.

The front-facing camera can capture an image of the user's face. Whilethe distance from the phone to the forearm is unknown, the distance fromthe phone to the face can be deduced from the pixel sizing of the interpupillary distance. (The closer the phone is to the face, the larger thedistance between the user's pupils becomes—in terms of pixel spacing.)Based on previous experimentation, or based on analysis of the camera'soptics, the pixel spacing between the depicted pupils directlycorrelates to the distance between the front-facing camera and theuser's face. Subtracting this value from 12 inches yields the viewingdistance between the smartphone and the user's forearm. From thisviewing distance, and information about the camera's optics, the size offeatures on the skin can be deduced.

Similarly, the color of facial skin depicted in imagery captured by thefront-facing camera, can be used in assessing the color of skin depictedin imagery captured by the rear-facing camera. In one scenario, thefacial skin may be used as a reference skin color. (Facial recognitiontechniques can be applied to identify the eyes and nose, and from suchinformation the portion of the imagery depicting cheeks and forehead canbe determined. Skin facial color can be sampled from these locations.)

Relatedly, eye color is a useful tool in establishing an expected skincolor. For example, a grey iris is most commonly associated with peopleof Northern and Eastern European descent, for whom norms of skincoloration can be established. Ethnic associations with other eye colorsare also well known. (See, e.g., the Wikipedia article “Eye color.”)

If imagery of the subject skin condition—captured by the rear-facingcamera—exhibits a skin color that is different than this referencecolor, such difference may be taken as a diagnostic indicia. Likewisethe reference facial skin color can be used in segmenting features fromthe skin imagery captured by the rear-facing camera.

In some instances, the skin imaged by the rear-facing camera (e.g., onthe user's forearm) may be illuminated differently than the facial skinimaged by the front-facing camera. For example, the user may haveoriented a fluorescent desk lamp towards their arm to provide morelight. As noted, such lighting changes the apparent color of the skin.Relatedly, the skin imaged by the rear-facing camera may be within ashadow cast by the phone. By comparing the skin colors imaged by thefront- and rear-facing cameras, such illumination issues can be detected(e.g., by difference in chrominance or luminance), and correctivecompensations then applied.

Longitudinal Studies

As noted, the evolution of a skin condition over time can be useful inits assessment. Images of a skin condition taken at different times canbe shown in different manners to illustrate evolution of the condition.

Desirably, the images are scaled and spatially aligned (i.e.,registered), so that a consistently-sized and oriented frame ofreference characterizes all of the images. This allows growth or otherchange of a lesion to be evident in the context of a generallyunchanging background.

Images can be scaled and aligned using known techniques. Exemplary is byreference to SIFT or SURF features, in which robust feature key pointsthat are common throughout images are identified, and the images arethen warped (e.g., by an affine transform) and rotated so that thesepoints become located at the same positions in each of the image frames.(One such arrangement is detailed in applicant's patent application20120208592.)

To facilitate this operation, it is desirable (although not essential)to first identify the extent of the lesion in each of the frames. Knownboundary-finding algorithms can be applied to this task (sometimespredicated on the assumption that the lesion of interest is found in thecenter of the image frame). Once the boundary of the lesion in eachimage is identified, the lesion can be masked (or flooded with a uniformcolor) so that the key point identification method does not identify keypoints from the lesion or its boundary. This reduces the key pointcount, and simplifies the later matching of common keypoints between theimages.

Body hair can also be a source of many superfluous key points in thedifferent image frames—key points that typically don't help, and mayconfound, the image registration process. Thus, the images are desirablyprocessed to remove hair before key points are determined. (There are avariety of image processing algorithms that can be applied for thistask. See, e.g., Abbas, et al, Hair Removal Methods: a Comparative Studyfor Dermoscopy Images, Biomedical Signal Processing and Control 6.4,2011, pp. 395-404 (attached to application 61/872,494), and referencescited therein.)

Key points are then extracted from the imagery. Depending on themagnification of the images, these points may be associated with nevi,hair follicles, wrinkles, pores, pigmentation, textures, etc. If theimaging spectrum extends beyond the visible, then features from belowthe outermost layer of skin may be evident, and may also serve as keypoints.

A key point matching search is next conducted to identify correspondingkey points in the images.

One image is next selected as a reference. This may be, e.g., the mostrecent image. Using the extracted key point data, the rotation andwarping required to transform each of the other images to properlyregister with the reference image is determined. These images are thentransformed in accordance with such parameters so that their key pointsspatially align with corresponding key points in the reference image. Aset of transformed images results, i.e., the original reference image,and the rotated/warped counterparts to the other images.

(If the lesion were on a flat, rigid surface, then each skin image wouldbe related to the others by a simple rotation and affine transform. Thisis generally a useful approximation for all cases. However, due to thecurvature of some skin surfaces, and the fact that skin may stretch, amore generalized transform may be employed to allow for suchvariations.)

One form by which the transformed images can be presented is as astop-action movie. The images are ordered by date, and renderedsequentially. Date metadata for each image may be visibly rendered in acorner of the image, so that the date progression is evident. Thesequence may progress automatically, under software control, or eachimage may be presented until user input (e.g., a tap on the screen)triggers the presentation to advance to the next image.

In some automated renderings, the software displays an image for aninterval of time proportionate to the date-span until the next image.For example, if images #1-4 were captured on successive Mondays, andthen two Mondays were missed before images #5-8 were captured (again onsuccessive Mondays), then images #1-3 may be presented for one secondeach, and image #4 may be presented for three seconds, followed byimages #5-7 presented for one second each. (Image #8—the last image—mayremain on the screen until the user takes a further action.) A userinterface control can be operated by the user to set the speed ofrendering (e.g., the shortest interval that any image is displayed—suchas one second in the foregoing example, or the total time interval overwhich the rendering should occur, etc.).

A different form by which the transformed image set may be viewed is asa transitioned presentation. In this arrangement, a video effectstransition is employed to show information from two or more image framessimultaneously on the display screen. In a simple arrangement, image #1(the oldest image) is displayed. After an interval, image #2 begins toappear—first as a faint ghosting effect (i.e., a low contrast overlay onimage #1), and gradually becoming more definite (i.e., increasingcontrast) until it is presented at full contrast. After a furtherinterval, image #3 starts to appear in like fashion. Optionally, theolder images can fade out of view (e.g., by diminishing contrast) asnewer images ghost-into view. At different times there may be data fromone, two, or more images displayed simultaneously. As before, theprogression can be under software, or user, control.

In the above examples, the renderings may employ the images from whichhair was digitally removed (from which key points were extracted).Alternatively, the renderings may employ the images with hairundisturbed.

In some arrangements, the rendering sequences can be accompanied bymeasurement data. For example, a textual or graphical overlay added to acorner of the presentation may indicate the width or area of thedepicted lesion, e.g., an area of 12 mm² in the first image, 15 mm² inthe second image, 21 mm² in the third image, etc. Similarly, for eachframe, the color or darkness of the lesion, or its boundary irregularityor its texture, may be quantified and expressed to the user.

In still other arrangements, such information is not presented with eachimage in the series. Rather, at the end of the rendering, information ispresented detailing a change in the lesion from the first frame to thelast (e.g., the lesion has increased in area by 83% in 7 weeks).

Such statistics about the lesion, and its changes, can also be presentedas a textual or graphical (e.g., with Cartesian graphs) report, e.g.,for emailing to the user's physician.

It will be recognized that the skin features from which the key pointsare extracted define a characteristic constellation of features, whichpermits this region of skin to be distinguished from others—afingerprint of the skin region, so to speak—and by extension, afingerprint of the user. Thus, even if a skin image is submitted to thecentral server data identifying the user, this characteristicfingerprint information allows the system to associate the image withthe correct user. This may be used as a privacy-preserving feature, oncea characteristic constellation of skin features has been initiallyassociated with a user. (This distinctive constellation of features canalso serve as a biometric by which a person can be identified—lesssubject to spoofing than traditional biometrics, such as friction ridgeson fingertips and iris pattern.)

It will further be recognized that features surrounding an area ofinterest on the skin effectively serve as a network of anchor points bywhich other imagery can be scaled and oriented, and overlaid, in realtime. This permits an augmented reality-type functionality, in which auser views their skin with a smartphone, and a previous image of theskin is overlaid in registered alignment (e.g., ghosted), as anaugmentation. (A user interface control allows the user to select adesired previous image from a collection of such images, which may bestored on the user device or elsewhere.) As the user moves the phonetowards or away from the skin—changing the size of the lesion depictedon the camera screen, the size of the overlaid augmentation similarlychanges.

As the size of the knowledge base increases, so does its utility. At asufficiently large scale, the knowledge base should enable detection ofpathologies before they become evident or symptomatic. For example, asubtle change in skin condition may portend a mole's shift to melanoma.Development of a non-uniformity in the network of dermal capillaries maybe a precursor to a cancerous growth. Signals revealed in skin imagery,which are too small to attract human attention, may be recognized—usingmachine analysis techniques—to be warning signals for soon-to-beemergent conditions. As imaging techniques advance, they providemore—and more useful—weak signals. As the knowledge base grows in size,the meanings of these weak signals become clearer.

To leverage the longitudinal information in the knowledge base, imageinformation depicting a particular user's condition over time must beidentifiable from the data structure. As described above, the uniqueconstellation of features associated with a particular region of skin ona user, allows all images depicting this patch of skin on this user tobe associated together—even if not expressly so-identified whenoriginally submitted. The FIG. 2 data structure can be augmented by afurther column (field) containing a unique identifier (UID) for eachsuch patch of skin. All records in the data structure containinginformation about that patch are annotated by the same UID in thisfurther column. (The UID may be arbitrary, or it may be derived based onone or more elements of user-related information, such as a hash of oneof the user's image file names, or based on the unique constellation ofskin feature points.)

As data processing resources permit, the central system can analyze thelongitudinal information to discern features (e.g., image derivatives)that correlate with later emergence of different conditions. Forexample, if the system finds a hundred users diagnosed with melanoma forwhom—in earlier imagery—a network of capillaries developed under a molethat later become cancerous, and this network of capillaries is denser,by a factor of two- to three-times, than the density of capillaries insurrounding skin, then such correlation can be a meaningful signal. If anew user's imagery shows a similar density of capillaries developingunder a mole, that user can be alerted to historical correlation of suchcapillary development with later emergence of melanoma. Such earlywarning can be key to successful treatment.

In the example just-given, the correlation is between a single signal(dense capillary development) and a cancerous consequence. Alsoimportant are combinations of signals (e.g., dense capillarydevelopment, coupled with die-off of hair in the mole region). Knowndata mining techniques (including supervised machine learning methods)can analyze the knowledge base information to discover such foretellingsignals.

Naturally, information discovered through such analysis of knowledgebase information is, itself, added to the knowledge base for future use.As new correlations are discovered, new insight intopreviously-submitted imagery may arise. The central system can issueemail or other alerts to previous users, advising them of informationthat subsequent data and/or analysis has revealed.

While the foregoing discussion concerned studies of a single skin site,it was earlier noted that information about skin conditions at otherplaces on the body may also be relevant. Thus, in conducting suchlongitudinal studies, consideration may also be given to information inthe knowledge base, and in the user data, concerning other skin sites.(Such other skin site information may be considered as another elementof user metadata, noted earlier, all of which should be employed indiscovering patterns of correlation.)

Application of machine learning technologies for cancer prediction is agrowing field of endeavor. See, e.g., Cruz, et al, Applications ofMachine Learning in Cancer Prediction and Prognosis, Cancer Infom., No.2, 2006, pp. 59-77; Bellazzi, et al, Predictive Data Mining in ClinicalMedicine—Current Issues and Guidelines, Int'l J. of Medical Informatics,V. 77, 2008, pp. 81-97; and Vellido, et al, Neural Networks and OtherMachine Learning Methods in Cancer Research, in Computational andAmbient Intelligence, Springer, 2007, pp. 964-971 (attached toapplication 61/872,494), and references cited therein.

Image Registration Using Multi-Spectral and Hyper-Spectral Imagery

The previous discussion, outlining approaches for registering images inlongitudinal studies, can benefit further when those images mightcontain more than just the standard RGB layers of a color image. Thereason for this is that skin texture, hair, lesion boundaries—and a muchlonger list of skin attributes than just these—all can be enhanced intheir discrimination and contrast by their representations as higherdimensional data structures. Many of the practical challengesencountered in trying to perform these registration techniques onstandard color imagery—such as feature point fading between one daywhere a patch of skin is drier and giving rise to distinct line featuresand the next day where a person has applied skin moisturizers and haseffectively removed those lines—can be overcome through the seeking andmeasurement of higher dimensional image features. The detailed warpingarrangements whereby stretched skin in one image must be re-stretched tomatch another image also can make use of such higher dimensionalhyper-spectral features.

Image Registration Through Motion

Another way to enhance skin-patch and/or lesion registration is throughteaching users of such an app to use and perfect the motion of theircamera itself while gathering imagery of their skin. The resultantmotion of the skin regions manifested in the imagery itself, oftencombined with the on-camera motion information data common to almost allsmartphones, allows for further degrees of information to be utilized inprecise millimeter and sub-millimeter scale matching of some given skinsample and the imagery of the same skin taken weeks, months or evenyears earlier. Many detailed features change often drastically over suchlengthy time periods, where the use of motion imagery and the resultantparallax data information can help to derive additional shape andperspective information, all of which assists in the generic task ofstabilizing the viewing and interpretability of often quite dynamic skinconditions and lesions.

Finger-Slide Viewing of Time Sequences

As is familiar in certain smartphone user interfaces, another method ofviewing skin conditions over time is to finger slide back and forth intime through the registered imagery. This doesn't need to be limited tothe viewing of pathological conditions either; it can be as simple asthe desire of a teenager to simply view how their make-up appearance haschanged over the last few weeks, or a review of what color of eye-linerthey might have utilized two weeks earlier. In such non-ailmentapplications of this technology, further user interface choices can bepresented to a user once they have honed in on some particular earlierimagery, such as displays of the very particular brands and types ofcommercial products may have an association with a particular earlierdate. Likewise, back in a pathological context, a user may be tryingvarious treatments for acne for example, and they will want to be ableto finger scroll between images taken when they were trying brand X, andimages when they we trying brand Y.

Managing Cultural Tensions Inherent in Automated Screening and ComputerAssisted Diagnosis (CAD)

Those familiar with the early 21st century growth in the use ofcomputers in helping to detect and diagnose disease are equally familiarwith the large divisions in the cultural acceptance of this inevitabletrend. It is difficult to disagree with the statement, “both sides areright.” If we caricaturize one camp as the proponents who correctlyclaim that computers can expand health care well beyond the wealthierclasses and countries, and the other camp as not necessarily opponentsbut critics who correctly claim that poorly executed health servicesoften violate the Hippocratic oath, then we find ourselves in astalemate that only time and the market will slowly break up.

This disclosure presents humble yet explicit technological componentsmeant to directly address these tensions as opposed to trying to ignorethem and wait for them to simply go away. That will take a while.Specifically, the numerous crowd sourcing aspects previously disclosedshould all be implemented with clear delineations between informationderived from clinically licensed sources versus everything else. Colorschemes, specially designed logos and text treatments . . . suchtechnically implemented graphic clues should all be used as templates todemarcate “results” information sent back to users of these systems. Asan example, if automated processes produce probability results whichtend to indicate concern over some skin patch or patches, anypositivistic results sent back to a user should be packaged in uniformlyrecognizable graphic forms which are associated with the classicresponse of “ . . . comparisons of your results with thousands of otherstend to suggest that you seek licensed medical examination . . . ” Onthe other extreme, if results tend strongly toward a “normal” or benignclassification for all submitted imagery, the graphic formatting andlanguage can indicate the null result, yet still reinforce the notionthat users should nonetheless use their instincts in seeking licensedmedical assistance despite the null results of some particular session.

These ideas can readily be implemented by considering them as a discretefilter stage sitting between the software/analysis engines that aretasked with producing probabilistic results on submitted imagery, andthe GUI stages of user interaction. This filtering is not at all a GUImatter; it is fundamentally about ensuring that centuries' old commonmedical practices are followed in the communication with a user. Thisdiscrete filtering will by no means solve the deep cultural tensionsinherent in these activities, but they can form a highly explicitbalance between the truths of both camps described above. The broadestgoal is to reach out to a broader set of actual at-risk individuals,providing guidance toward the seeking of licensed medical treatment.Likewise, for those individuals who in actuality are not at risk for theconditions they are worried about, it is still no place for an automatedservice to do anything more than simply indicate null results. No traceof “assurances” can be part of a response unless a licensed practitioneris actively involved in a session, with full disclosure of thatinvolvement and reference to the professional acceptance of thatinvolvement. All in all, then, this discrete filter might be named the“best medical practices” filter, and all communications concerning testresults should be mandated to pass through this discrete filter.

The classic term “screening” has been used for decades now, largelydealing with these broader concepts. Though the lay-public oftenconfuses screening with diagnosis, the medical profession has put forthenormous efforts to educate the public about their differences.Furthermore, many medical professionals will consider any automatedservice which does not have a case-by-case medical practitioner involvedto not even be worthy of the term “screening.” This is a legitimateviewpoint, especially as the law allows for any solution vendor ormedicinal product manufacturer to make generic claims toward medicalefficacy. But here again the term “screening” can become a technicallyimplemented element of the crowd-sourced elements described in thisdisclosure by simply presenting results in the fully disclosed contextin which those results were derived. Specifically, deliberatelyborrowing from known cultural norms, results can be phrased as “over1000 other individuals have submitted images very similar to yours, andaccording to an ad hoc survey of those individuals, 73% sought medicaladvice . . . ” The variations and permutations on these themes are vast.

Again, technically, this kind of data and the generation of suchstatements require actual crowd-source data gathering and storage, thenlinked into a results filter as described above. The actual process ofthis particular type of screening is fully disclosed both in itsmethodologies as well as in the phrasing of results. It thus earns theterm screening because that's exactly what it becomes, a crowd-sourcedscreening phenomena. Its eventual efficacy will be determined by thequality of its ultimate results and growth in its user base.

Quick and Economic Whole-Body Skin Screening—Early Melanoma ScreeningRoom

Fresh off the topic of screening, this disclosure next details how thecurrent art of whole-body dermatological photography can transitiontoward a fully licensed dermatological screening test, emulating thecultural norms of pap smears and colonoscopies.

The current art in whole-body scanning is illustrated by CanfieldImaging Systems, which operates imaging centers in cities throughout theU.S. At these facilities, patients can obtain whole-body imagery, whichis then passed to their physicians for review. Aspects of the Canfieldtechnology are detailed, e.g., in documents U.S. Pat. Nos. 8,498,460,8,218,862, 7,603,031, and 20090137908.

A basic aspect of an illustrative early melanoma screening room (EMSR)is simple: build a transparent phone booth (or cylinder) surrounded withcameras and synchronized lighting. For example, the booth may compriselighting with, e.g. 16 to 32 LED spectral bands (some into the near-IR),and a dozen or two dozen RGB and/or black and white cameras. (See, e.g.,patent application documents 20130308045, and Ser. No. 14/201,852.)Shaving, or an alcohol or other skin treatment, may be employed incertain cases. People get naked or put on a bathing suit, get inside,and raise their arms—as in the TSA imaging booth. Five or fifteenseconds later they are done. They can wear small goggles if they like,but closing eyes, or even having them open, will probably be fine (andwill probably have to be, for FDA approval). Maybe two or three posesfor normal extra data gathering, dealing with odd reflections and glare,different skin-surface normals, so all in all a non-surgical,less-than-one-minute affair.

The computer churns for another 30 seconds conducting image analysis andcomparison with reference data, and either gives a green light, or,perhaps in a non-alarming way and still “routine,” a low threshold isset such that a patient is asked to go into a second room where atechnician can focus in on “concern areas” using existingstate-of-the-art data gathering methods on exact areas, including simplescrape biopsies. (The technician views results from the scan, with“guidance” from the software, in order to flag the patient, point outthe areas of concern, and instigate the second-room screening.)Practicing clinicians also can be more or less involved in the steps.

This is pap-smear, colonoscopy, cultural 101 kind of thinking . . . doit first when you are 25, then every 5 years, or whatever. Get it closeto the pap-smear kind of test cost wise, the rooms themselves shouldn'trun over $5 to $10K full manufacturing cost; no need to get too crazy onthe hardware technology.

The market demand, of course, is to discriminate normal skin frommelanoma and other pathologies. The explicit target would be detectingearlier and earlier stage melanoma, seeing how early one can get.Receiver operating characteristic (ROC) curve studies are the industrynorm next step, seeing how quickly true positive detections can occurbefore annoying levels of false positives start to kick in. Since thisis meant to be a very early screening method, this favors tilting theROC curves toward “detect more,” so that again, a technician can do “nobig deal” secondary screening using existing methods and weed out theslightly larger level of false negatives due to higher ROC thresholds.So this is also a cost-based measure, providing better guidance toward“who” should be getting referred to more expensive existing screeningmethods.

The real point of EMSR is early detection. Increased quantity of care atcurrent quality levels and current cost levels is also the point, withaverted mortality also being a direct cost benefit beyond saving aperson's life. Even six months, but better yet 1 to 2 years of advanceddetection will produce stunning and clear increases in survival rates.

FIG. 5A shows a schematic sectional view looking down into a cylindricalbooth (e.g., seven feet in height, and three feet in diameter, with anaccess door, not particularly shown). Arrayed around the booth (inside,outside, or integrated into the sidewall) are a plurality of lightsources 52 and cameras 54.

The depicted horizontal ring array of light sources and cameras canrepeat at vertical increments along the height of the booth, such asevery 6 or 18 inches (or less than 6, or more than 18). The lights andcameras can align with each other vertically (i.e., a vertical linethrough one light source passes through a series of other light sourcesin successive horizontal rows), or they may be staggered. Such astaggered arrangement is shown in FIG. 6A, in which successive rows oflights/cameras are offset by about 11 degrees from each other.

FIG. 6B shows another staggered arrangement, depicting an excerpt of theside wall, “unwrapped.” Here, successive horizontal rows of lightsources 52 (and cameras 54) are offset relative to each other. Moreover,in this arrangement, the light sources 52 are not centered betweenhorizontally-neighboring cameras, but are offset.

Although not depicted, the light sources needn't be interspersed withcameras, with the same number of each. Instead, there may be a greateror lesser number of light sources than cameras.

Similarly, the light sources needn't be arrayed in the same horizontalalignment as cameras; they can be at different vertical elevations.

Light sources and cameras may also be positioned below the person, e.g.,under a transparent floor.

Desirably, the light sources are of the sort detailed in applications20130308045 and Ser. No. 14/201,852. They may be operated at asufficiently high rate (e.g., 40-280 Hz) that the illumination appearswhite to human vision. The cameras transmit their captured imagery to acomputer that processes the images according to methods in thejust-noted patent documents, to determine spectricity measurements(e.g., for each pixel or other region of the imagery). Desirably, errorsin these measurements are mitigated by the techniques detailed in thesedocuments. The imagery is then divided into patches (e.g., 1, 2, or 5 cmon a side) and compared against reference imagery, or applied to anotherform of classifier. All of the imagery can be processed in this fashion,or a human or expert system can identify patches of potential interestfor analysis.

In other embodiments, the patient can stand on a turntable that rotatesin front of a lesser number of cameras and light sources, while framesof imagery are successively captured. Thus, a full “booth” is notrequired. Such arrangement also captures imagery at a range of differentcamera-viewing and light-illuminating angles—revealing features that maynot be evident in a static-pose capture of imagery. (A turntable alsoallows hyperspectral line sensors to be employed in the cameras, with 2Dimagery produced from successive lines as the turntable turns. Such linesensors are available from IMEC International of Belgium, and capture100 spectral bands in the 600-1000 nm range. Of course, 2D sensors canbe used as well—including hyperspectral sensors. One vendor ofhyperspectral 2D sensors, sometimes termed imaging spectrographs, isSpectral Imaging Ltd. of Finland.)

In some embodiments, 3D reconstruction techniques (e.g., SLAM) areapplied to the captured imagery to build a digital body map. In such amap/model, not only the size, but images, of every suspicious locationare recorded over time. In some implementations, images of the entirebody surface can be recorded, allowing an examining physician tovirtually fly, Google-Earth-like, over the patient's modeled bodysurface, pausing at points of interest. If historical images areavailable, the physician can examine the time-lapse view of changes ateach location, as desired. Some useful subset of the spectral bands canbe used to do the mapping. If desired, the patient's body map can bemorphed (stretched and tucked and squeezed, etc.) to a standardized 3Dbody shape/pose (of which there may be a dozen or more) to aid inautomated processing and cataloging of the noted features.

User Interface and Other Features

In reporting results from crowd-sourced data repositories back to users,care should be taken not to offend. In one aspect, the user softwareincludes options that can be user-selected so that the system does notpresent certain types of images, e.g., of genitalia, of morbidconditions, of surgical procedures, etc. (Tags for such imagery can bemaintained in the knowledge base, so that images may be filtered on thisbasis.)

The user interface can also allow the user explore imagery in thedatabase. For example, if the system presents a reference imagedepicting a leg lesion that is similar to a lesion on the user's leg,the user may choose to view follow-on images of that same referencelesion, taken at later dates—showing its progression over time.Similarly, if the reference lesion was found on the leg of a prior userwho also submitted imagery showing a rash on her arm, the current usermay navigate from the original leg lesion reference image to view thereference image showing the prior user's arm rash.

Image navigation may also be based on image attribute, as judged by oneor more parameters. A simple parameter is color. For example, onederivative that may be computed for some or all of the images in theknowledge base is the average color of a lesion appearing at the middleof the image (or the color of the pixel at the middle of the image—if ageneralized skin condition such as a rash is depicted there). The usercan query the database by defining such a color (e.g., by pointing to alesion in user-submitted imagery, or by a color-picker interface such asis employed in Photoshop software), and the software then presents theimage in the knowledge base that is closest in this metric. The user mayoperate a control to continue such exploration—at each step beingpresented an image that is closest in this attribute to the one beforeit (but not previously displayed).

Similarly, the user interface can permit user navigation of referenceimages based on similarity in lesion size, shape, texture, etc.

Hair on skin can be a useful diagnostic criterion. For example, melanomais aggressively negative for hair; hair is rarely seen from suchgrowths. So hair depictions should be included in the knowledge baseimagery.

However, hair sometimes gets in the way. Thus, certain of the processingmay be performed using image data from which the hair has been virtuallyremoved, as detailed earlier.

The user interface can allow the user to tap at one or more locationswithin a captured skin image, to identify portions about which the useris curious or concerned. This information is conveyed to the centralsystem—avoiding ambiguity about what feature(s) in the image should bethe focus of system processing. The user interface can allow the user toenter annotations about that feature (e.g., “I think I first noticedthis when on my Las Vegas vacation, around May 20, 2013”).

Additionally, or alternatively, when the central system receives a userimage, and processes it against the knowledge base information, it mayreturn the image with one or more graphical indicia to signal what ithas discovered. For example, it may add a colored border to a depictedlesion (e.g., in red—indicating attention is suggested, or in green), orcause an area of the screen to glow or strobe. When this image ispresented to the user, and the user touches or otherwise selects thegraphical indicia, information linked to that feature is presented,detailing the system's associated findings. A series of such images—eachwith system-added graphical indicia (e.g., colored borders)—may berendered to illustrate a time-lapse evolution of a skin condition, asdetailed earlier.

Skin is our interface between our body and our world; our interactionwith our environment is largely recorded on this thin layer. The presenttechnology helps mine some of the wealth of information that this recordprovides.

Rainbow Mode

Reference was made to gathering multiple frames of imagery underdifferent, spectrally tuned illumination conditions, such as by usingillumination sources (e.g., LEDs) tuned to different wavelengths.

As detailed in cited publication 20130308045 and application Ser. No.14/201,852, N different spectral illumination sources combine with Mdifferent spectral detectors (e.g., the three different color filtersoverlaying a smartphone photodetector array) to yield up to N*Mdifferent sets of image data. From this richness of different imagedata, a rich set of different features can be discerned.

Those patent applications further detailed how ambient lighting effectscan be largely removed, even if the spectrally-tuned illuminationamounts to just 10% or so of the total illumination.

In accordance with a further aspect of the present technology,illumination at different spectral wavelengths is provided byillumination from a smartphone screen. One such method captures imageryusing a smartphone's front-facing camera (i.e., the camera on the sameside of the phone as the touchscreen), instead of the usual rear-facingcamera. The field of view captured by the front-facing camera is thenilluminated—at least in part—by light from the smartphone screen. Thisscreen is software-controlled to present a sequence of differentillumination patterns or colors (“rainbow mode”), during which differentframes of imagery are captured.

Smartphone and other screens commonly emit “red,” “green” and “blue”light—each with a particular spectral profile. (This profile typicallyvaries from one type of smartphone to another—due to different displaytechnologies, and sometimes varies among smartphones of the same typedue to process variations.) Importantly, these spectral profiles neverexactly match the red-, green- and blue-Bayer sensor pixel spectralprofiles—giving rise to the multiplicative effect noted above.

For example, some “blue” illumination from the display screen willslightly “light up” green Bayer-filtered pixels. “Green” illuminationfrom the display screen will excite all three (R/G/B) Bayer-filteredphotosensors. “Red” from the display will excite both red- andgreen-filtered photosensors. This example gives sevencross-combinations, or “channels,” which the cited patent disclosuresdiscuss in great detail, e.g., describing and characterizing how thesechannels vary in quality and information content. (While there are ninecombinations of screen colors and photosensor colors, blue and red aresufficiently remote—spectrally—that the coupling effect is essentiallynil.)

OLED displays are coming into widespread use (e.g., the Samsung GalaxySIII, S4 and S5) and offer increased brightness and wider gamut,compared with previous technologies.

Flexible displays are also beginning to appear commercially. These areattractive to illuminate 3D relief features at close imaging distances,as the display is operated to sweep illumination from one side to theother.

Autostereoscopic displays (commonly including parallax barriers) canalso be used, and can create structured illumination.

One illustrative embodiment uses rainbow mode in capturing andprocessing frames of image data from a user's face. In a particularimplementation (which may be called Face-Chek), the motion and/or poseof a smartphone is sensed, and used to switch between data collectionand data presentation modes.

As is familiar, the motion and pose of a smartphone can be discerned byreference to data from the phone's onboard accelerometers, magnetometersand gyroscopes—each commonly 3D. (Motion can also be assessed byreference to apparent movement of imagery captured by the phone camera.)

In data collection mode, the user waves the phone around the head (andoptionally scalp), capturing frames of imagery with the front-facingcamera, from different vantage points. During such motion, the phonescreen is displaying a sequence of different illuminationpatterns/colors. Software analyzes the imagery captured under thesevarious illumination/viewpoint conditions to identify features ofpotential concern, classify features by type, create longitudinal imagesequences illustrating changes in particular features, etc. When thedevice is thereafter held static by the user for viewing, thisinformation is presented. (This static pose is typically one in whichthe screen is inclined upwardly, with the base of the screensubstantially horizontal, i.e., within ten degrees of the horizon).

The screen illumination can be of various types. At some update rates,human persistence of vision causes the illumination to seem uniform,e.g., all white. At slower rates, different colors or patterns flashacross the screen.

A simple arrangement sequentially displays screens of all-red,all-green, all-blue, in cyclical fashion. In a variant, the phone's“torch” (i.e., illumination flash) is operated in a fourth phase of thesequence, giving four different illumination states.

In another arrangement, solid screen colors are still employed, but thistime with combinations of the red/green/blue primaries (yielding whatmay be termed cyan, magenta and yellow).

In some arrangements, the transitions between colors are abrupt. Forexample, a red screen can be maintained for a sixth of a second, andthen switch to blue for the next sixth of a second, etc. In otherarrangements, the transitions are blended. For example, a displayedsolid color may be updated thirty times a second. At a first frame, redis presented. At the second frame, 20% of the red pixels are changed togreen. And at the third frame, 20% more of the red pixels are changed togreen. A seemingly-continuous smear of colors results (but is actually15 different colors. Twice a second the display is all-red. Ditto forall-blue and all-green.

Capture of image frames by the camera is synchronized to the differentframes of illumination. In the example just-given, the camera maycapture six frames of imagery per second (i.e., two with all-redillumination, two with all-blue illumination, and two with all-greenillumination).

Skin topology features are best revealed by illuminating the skinobliquely, at various different angles. This can be done by operatingthe screen to present illumination from different parts thereof, atdifferent times. The rest of the screen can be kept dark (black).

One such arrangement is shown in FIG. 7. The display screen at the topof the figure is all-dark (black). After a fixed interval a colored bandappears along the left edge of the screen. A further interval later itshifts one band-width to the right. In similar fashion the color bandmarches across the screen. (While six discrete steps are illustrated, agreater or lesser can be used.)

After the colored band has finished its march across the screen, a darkscreen is then presented, followed by a similar march of a band of adifferent color.

In some embodiments, the sequence of colors (e.g., of the FIG. 7 bands)is sequential, e.g., red, green, blue (or red, green, blue, cyan,magenta, yellow, etc.). In others, the sequence of colors is random.

Likewise with the FIG. 7 progression of the colored band from left toright. In some arrangements this pattern is repeated, with differentcolors. In others, the direction of the band's movement is changed fromone cycle to the next. For example, after the red band of FIG. 7 hasmarched to the right of the screen in one cycle, the next cycle may havea band of color march from the top to the bottom. Or from the right tothe left. Or from the bottom to top. Again, the sequence can repeat, orrandom directions and/or colors can be used.

FIG. 8 shows a variant in which two bands—of different colors—moveacross the screen. As in the arrangement just-described, the color ofthe horizontal band in different cycles can follow a repeating pattern,or it can be selected randomly. Similarly with the direction of thehorizontal band's movement (top-to-bottom, or bottom-to-top). Likewisewith the vertical band.

FIG. 9 shows yet another variant, in which blocks of different colorsappear at different positions on the screen. As before, the sequence ofcolors, and positions of the blocks, can follow a repeating pattern, oreither/both can proceed in a random sequence.

(In this and other embodiments discussed herein, the device can be aniPhone 5, which has a display that is four inches in diagonalmeasurement, and has a 1136×640 pixel resolution, i.e., 326 points perinch.)

FIG. 10 further considers certain aspects of the illumination geometry.At the top of the figure, a color band is displayed at one end of thesmartphone screen. Light from this band illuminates a location 102 onthe skin at an angle of 21 degrees (relative to a tangent 104 to theskin surface).

A few instants later, the band has moved to near the middle of thescreen, as depicted in the middle of the figure. Here, light from thisband (which may now be of a different color, or not) illuminates theskin location 102 at an angle of 40 degrees.

A few instants later, the band has moved to hear the opposite end of thescreen, as depicted in the bottom of FIG. 10. Here, light from this bandilluminates the skin location at an angle of 103 degrees.

As before, the phone captures a frame of imagery under each of thesedifferent illumination conditions, using a camera 106.

It will thus be recognized that the FIG. 10 arrangement collects imageryof skin location 102 as illuminated from a variety of different angles.If the color of the band changes as it marches across the display, sucharrangement collects imagery of the skin location with illumination thatis diverse both in angle and spectrum.

While the phone is stationary in FIG. 10, in actual practice, the phonemay be moved by the user to different positions relative to skinlocation 102 while the color band marches across the display screen.Thus, in some embodiments, imagery is captured having diversity inillumination angle, illumination spectrum, and viewpoint.

(To fit three illumination scenarios in the FIG. 10 illustration, thephone is depicted as closer to the skin than may generally be the case.A more typical imaging distance is 4-6 inches.)

The oblique illumination afforded by colors presented on different partsof the screen, together with motion that moves the camera to differentviewpoints, helps in revealing millimeter-scale 3D profiles of the skin.Such profile information can aid in classifying certain skin conditions,e.g., dryness, scaliness, age wrinkles, sunburn, etc.

By producing a sequence of, for example, 30 image frames over a onesecond period, where, for example, 10 different states of rainbowillumination were occurring over that one second period, an informationdiversity situation is created that goes beyond just the spectralcombinations of the display RGB and the sensor RGB; illumination angleeffects are now also included in this large gathering of imagery data.As is well known in machine learning and the wide field ofclassification of objects, it may not always be possible to analyticallyunderstand this larger complexity represented in the data. However, thevery existence of this extra complexity will allow for higherdimensional discrimination of one class of “thing” over another class of“thing.” In the case of this technology and skin, this can involve thesimplest of questions such as “is this skin drier or more moist thanthat skin?” The way this is done is through classic training, whereknown samples of drier skin, and known samples of more moist skin, allget sampled via these rainbow illumination ways, and their resultanthigher dimensional differences and unique characteristics all get“trained in” to this otherwise incomprehensible set of data. Modernclassification approaches routinely deal with data vectors having 100 oreven 1000 or more dimensions of related information. A patch of skinhaving been illuminated with all these spectral combinations and fromthese various spatial directions can easily give rise to dozens orhundreds of bundled data attributes at millimeter by millimeter scalesrepresented by the pixels of the camera. Spatial derivatives of thesedata, e.g., how does the “red light from the left pixel datum” changefrom this pixel to ten pixels over, innately carry discriminationinformation between a new red rash that might be lumpy versus the lesslumpy nature of a hickey, to use a deliberately out-there example. Thus,perhaps, boyfriends and girlfriends can quickly check for any youthfulinfidelities as but one tiny application for rainbow-derived datacaptures.

Sunburn analysis is another application where spatial-spectral rainbowdiversity can enrich the identification and discrimination power ofcaptured imagery. The emergence of even very small blistering, perhapsvery difficult to see even with a magnifying glass, will neverthelessexhibit different far-field optical characteristics over, say,non-blistering but nevertheless still-painful lesser degree burns.Dryness detection of skin was already mentioned, whereby with rainbowdiversity, it is possible to build up a “scaling system” of degrees ofdryness, such that users can make their own choices about when and whennot to apply various skin moisturizers. The aging of skin is anotherarea where there are a vast number of people interested in betterunderstanding not simply what to do about aging skin, but to understandthe state of their own skin and to understand how they might be able totreat and track the various remedies they employ in the battles againstaging. The rainbow spectral-spatial diversity here again can provide aricher set of direct empirical samplings of the intricacies of widevarieties of skin conditions which, after all, derive from a verycomplicated three-dimensional structure, namely human skin.

The general principles of shining spectrally shaped light from someangle in space can of course be accomplished by LED's that are somewhatcollimated or even tunable lasers. The broadest principle being taughthere is that though the microbiology of specific skin conditions or skintopologies may be too complicated to create recognizable patterns fromthis diversity of lighting angles and spectra, it nevertheless can stillbe measured and used to discriminate condition A from condition B, oreven stages of progression of some singular condition. As machinelearning practitioners understand, the resulting data vector populationsdisperse themselves in higher dimensional vector spaces in such a waythat by training on known condition types, generating signatures of thepopulations a la SVM types of boundary conditions on these types, thencomparing unknown test cases against these populations can be a methodfor discriminating conditions which otherwise might look nearlyidentical to a normal color camera not employing these rainbow samplingtechniques.

(Additional information regarding acquiring data from multiple-viewpointimagery, e.g., to detail surface attributes, is detailed in applicant'spublication 20080112596.)

Audio, Etc.

In accordance with other aspects of the present technology, audio (andother physiologic data) is collected and used in manners like the skinimagery herein.

For example, one particular embodiment employs regression analysis on aset of audio data to characterize false conclusions that should not bedrawn. (Culling the false helps in identifying the truth.) Anothercompares extracted features against templates of “normal” features, toidentify anomalous signals that should be reviewed by a qualifiedphysician.

Patent publications 2002085724, 2005078533, 2008046276, 2008062389,2010045210, 2011009759, 2011021939, 2011096820, 2011076328, 2012043018,2012090303, 2013016584, 2013061484, 2014012149, WO0209396, andWO13156999 detail a variety of technologies useful in collecting andprocessing physiological sound information, which are suitable for usewith the arrangements described herein.

Among the different audio data collection arrangements are bell anddiaphragm type pick-ups, acoustically coupled to one or moremicrophones. Such a pickup can be provided in a separate unit—coupled toa portable device (e.g., smartphone) by wire (e.g., employing theheadphone/mic jack) or wirelessly (e.g., employing Bluetooth).Alternatively, such a pickup can form part of the portable device,either permanently, or by an accessory unit that is removably attachedto the device body. An example of the latter is a downwardly-openingfunnel-like member (e.g., made of plastic) that friction-fits over thelower inch or half-inch of a smartphone body, channeling sounds from thewide end of the funnel up to the microphone(s) normally used fortelephone communication.

In other arrangements audio sensing can be done by worn microphones. Insome embodiments, one or more microphones are provided in a band (e.g.,a wrist- or waist-band) worn by a user (or worn by a clinician, forprobing a user). In others, acoustic sensors are integrated in clothingor other garments. By positioning microphones at different locations onthe body, the spatial origins of different sounds can be betterdetermined, aiding their diagnostic significance. In still otherarrangements, head-worn microphones can sometimes be employed (e.g.,Google Glass-like arrangements).

To reduce the influence of spurious (i.e., non-physiologic) audio foundin the ambient environment, one or more other microphones can beemployed to sense the ambient audio, so it can be removed from thesensed physiologic audio, using known noise cancellation techniques.(Smartphones are increasingly provided with multiple microphones; one ormore of these can be used to enhance the sensed physiological signals.)

A variety of diagnostically relevant acoustic signals can be sensed.These include heart sounds and other cardiovascular sounds (includingmurmurs, bruits, and other blood flow noises), lung and otherrespiratory sounds (including crackles, rales, rhonchi, wheezes, coughs,snoring and other air flow noises), bowel and digestive sounds, jointnoises (e.g., pops and creaks), as well as speech and othervocalizations.

In gathering data for analysis, and for comparison against referencedata, a variety of processing techniques can be employed.

One processing technique characterizes and strips-out positioningsounds, e.g., when a microphone is moved and rubbed against a person'sskin, before settling at a final position for data collection. Aclassifier can be trained to recognize such positioning sounds, so thatthey are not used in diagnostic processing.

Another processing technique is noise-cancellation, e.g., as notedabove, and in certain of the cited patent documents. A suitablewavelet-based denoising arrangement is detailed in Messer, et al,Optimal Wavelet Denoising for Phonocardiograms, Microelectronics Journal32.12, pp. 931-941 (2001). Spectral filtering can also be employed, whensame is desired based on measurement context.

Other signal processing techniques useful with physiologic signals maybe broadly classified into the following overlapping areas:

-   -   1. Prediction based techniques—useful for cardiovascular and        respiratory sounds that have a predictable (repetitive)        structure        -   a. Prediction error can be useful in detecting abnormalities            in heart beats and also in isolating noise-like signals,            e.g., a murmur, in the presence of a stronger heart beat            signal.        -   b. Transient signals are not easily predictable either. So,            in analyzing a wet cough signal, there may be a transient            component which will be present in dry cough but in wet            coughs there will also be a predictable component which will            be absent in the dry coughs (based on an excitation model            for coughs).    -   2. Time-frequency techniques—FFTs, Spectrograms etc.        -   a. Spectrograms can be analyzed for high frequency vs low            frequency signals—noise-like signals have higher frequency            content—air flow sounds, murmurs etc.    -   3. Transient analysis—short frame analysis        -   a. Pops and crackles are transient signals and they can be            detected/analyzed using short frame audio analysis        -   b. Energy envelope is also useful in transient signal            analysis    -   4. Fingerprinting techniques        -   a. With enough training examples, audio fingerprinting            features and matching techniques from the field of music and            video recognition can work for physiologic sounds. (Examples            of audio fingerprinting and matching are detailed in patent            publications 20070250716, 20070174059 and 20080300011            (Digimarc), 20080276265, 20070274537 and 20050232411            (Nielsen), 20070124756 (Google), U.S. Pat. No. 7,516,074            (Auditude), U.S. Pat. No. 6,990,453 and U.S. Pat. No.            7,359,889 (Shazam), 20050141707, 20050259819, and            20060075237 (Gracenote), and U.S. Pat. No. 8,656,441 (Cox).)        -   b. One example of a fingerprinting based diagnosis is to            collect “healthy” audio data (when the patient is healthy)            and then analyze any deviation from the signatures or            fingerprints of this healthy data to determine pathologies            in future diagnostic examinations.

Such processing can provide a variety of “features” that can be comparedwith reference data in assessing whether a signal is normal or anomalous(and, if the latter, used to help identify what the anomaly is—or isnot).

One approach is to use an auto-regressive model to parameterize thesensed sounds. This is the approach employed, e.g., by Harma et al inTime-Varying Autoregressive Modeling of Audio and Speech Signals, Proc.of the EUSIPCO, pp. 2037-2040, 2000.

Another approach uses features including the MFCC (Mel-FrequencyCepstral Coefficients) and PLP (Perceptual linear predictor) that areused in the automatic speech recognition field.

Coughs, snores, bruits, and other isolatable or cyclically recurringsounds can be parameterized, in one respect, by identifying the timeinterval over which a threshold (e.g., 80%) portion of the spectralenergy is expressed, and the smallest frequency bandwidth (characterizedby low and high frequencies) within which a threshold (e.g., 90%)portion of the spectral energy is expressed. (These low and highfrequency bounds can also independently serve as useful features.)

A large number of other parameters and features can similarly be derivedfrom a spectrogram of the audio, as illustrated by the following Table1, adapted from the thesis by Glaeser, Analysis and Classification ofSounds Produced by Asian Elephants (Elephas Maximus), Portland StateUniversity, 2009. (In this table, the just-noted features areessentially features M5, M6, M3 and M4, respectively.)

TABLE 1 Parameter Units Description Type M1-M6: Feature box Start Timesec Lowest time index in bounds that encompass the temporal (M1) inner90% of the signal strength in the time envelope. End Time sec Highesttime index in bounds that encompass the temporal (M2) inner 90% of thesignal strength in the time envelope. Lower Hz Lowest frequency index inbounds that frequency Frequency encompass the inner 90% of the signalstrength in (M3) the frequency envelope. Upper Hz Highest frequencyindex in bounds that frequency Frequency encompass the inner 90% of thesignal strength in (M4) the frequency envelope. Duration sec Width offeature box: M2-M1 temporal (M5) Bandwidth Hz Height of feature box:M4-M3 frequency (M6) M7-M14: Central values and variation Uses measuresthat do not assume normality: median, quartile ranges, quartileskewness, concentration. Median Time sec Time at which 50% cumulativesignal energy is temporal (M7) reached. (Measured relative to start ofsound, so M7 was calculated as M7new = M7 − M1) Temporal secConcentration of a sound around the median time temporal Interquartile(M7) measured as the duration of the interquartile Range range of signalenergy (Q3-Q1). (M8) Counts energy going forward and back from themedian time (M7). Q3 = median + 25% of signal energy Q1 = median − 25%of signal energy Temporal sec Concentration of a sound measured as thetime temporal Concentration span encompassing loudest 50% of timeenvelope (M9) values. Counts energy from the loudest parts down towardsthe smallest parts regardless of where the parts occur in time. Temporalnone Skewness of energy along time axis within temporal Asymmetryinterquartile range (−1.0 to 1.0) (M10) Median Hz Frequency at with 50%cumulative signal energy frequency Frequency is reached. (M11) Morestable than extreme values of LowerFreq and UpperFreq in varying noiseconditions. Spectral Hz Concentration of a sound around the medianfrequency Interquartile frequency (M11) measured as frequency range ofRange interquartile range of signal energy (Q3-Q1). (M12) Counts energygoing forward and back from the median frequency (M11). Q3 = median +25% of signal energy Q1 = median − 25% of signal energy Spectral HzConcentration of a sound measured as the frequency Concentrationfrequency span encompassing loudest 50% off (M13) requency envelopevalues. Counts energy from the loudest parts down towards the smallestparts regardless of where the parts occur in time. Frequency noneSkewness of energy along frequency axis within frequency Asymmetryinterquartile range (−1.0 to 1.0) (M14) M15-M20: Peak intensity Time ofPeak sec Time of single loudest spectrogram cell. temporal CellIntensity Time of the cell containing the peak intensity. (M15)(Measured relative to start of sound, so M15 was calculated as M15new =M15 − M1) Relative % Relative time of peak intensity (M15/M5) temporalTime of Peak Cell Intensity (M16) Time of Peak sec Largest value in timeenvelope, which is the temporal Overall largest vertical sum of thespectrogram over all Intensity frequencies. Time of the peak intensityin the (M17) trimmed time envelope. (Measured relative to start ofsound, so M7 was calculated as M17new = M17 − M1) Relative % Relativetime of peak intensity (M17/M5) temporal Time of Peak Overall Intensity(M18) Frequency of Hz Frequency of cell containing the peak intensity.frequency Peak Cell Intensity (M19) Frequency of Hz Frequency of peakintensity in the trimmed frequency Peak Overall frequency envelope.Intensity (M20) M21-M24: Amplitude and frequency modulation (variationof amplitude and frequency over time) AM Rate Hz Dominant rate ofamplitude modulation. amplitude (M21) Frequency of the maximum rate inthe power spectrum of the trimmed time envelope. AM Rate Hz Variabilityof amplitude modulation measured as amplitude Variation the width ofpeak at M21-6 dB. (M22) Values are discretized because at 6 dB down fromthe peak, the widths may be a only a few bins wide so the values areinteger multiples of the bin width. FM Rate Hz Dominant rate offrequency modulation. frequency (M23) Frequency of the maximum rate inthe power spectrum of the trimmed frequency envelope. FM Rate HzVariability of frequency modulation measured as frequency Variation thewidth of peak at M23-6 dB. (M24) (How much the rate of change varies,may be related to inflections and steepness of upsweeps and downsweeps)M25-M28: Fine features of harmonic structure, shifts in periodicity,frequency change, rate of change in frequency direction of Cepstrum HzHarmonic structure structure Peak Width Average width of peaks(harmonics) in power (M25) spectrum. Peak width is measured at 6 dB downfrom maximum value. At 6 dB down from the peak, the widths may be a onlya few bins wide (like M22 and M24), but M25 is an average of integers sothe values are not discretized. Narrow peaks means narrowband/tonalharmonics. Overall Hz Entropy, shifts in periodicity structure EntropyDistribution of energy across frequency blocks in (M26) a given timeblock. Shift from periodicity and linearity to chaos. Change innoisiness v. tonality. Upsweep Hz Direction of frequency changefrequency Mean Measures how much the frequency increases. (M27) Averagechange in median frequency between successive time blocks, weighted bytotal energy in the block. Inflection points with rising and fallingfrequencies throughout call result in a low M28 (closer to 0) comparedto a consistent directional change. Measure is weighted to emphasizecontribution of louder signal components. M27 < 0 means frequency isdecreasing. Upsweep % Rate of directional frequency change frequencyFraction Counts the number of times the frequency content (M28)increases. Fraction of time in which the median frequency in one blockis greater than in the preceding block, weighted by total energy in theblock. Indicates how much of the call has a directional change in thefrequency. Inflection points with rising and falling frequenciesthroughout call result in a high M28, just as a consistent directionalchange. Measure is weighted to emphasize contribution of louder signalcomponents. M28 always positive. M29: Signal strength Signal-to- dBSignal to noise ratio within the sampled sound. amplitude Noise RatioRatio of the signal power (loudest cell) to the (M29) noise power (powerof cell at 25^(th) percentile). Cells are ranked low to high and thecell at the 25^(th) percentile represents noise. (25^(th) percentile isused because the sound likely takes up less than 75% of the totalspectrogram cells.) Measurement assumes that the within the annotationbox at least 25% of the cells are without a focal signal.

An ensemble of some or all of the twenty-odd parameters just detailed(as well as others) can be computed for a sensed sound, and comparedwith reference data to discern similarity to (or deviation from)reference ensembles of parameters—either individually, or in statisticalfashion (e.g., a particular respiratory sound is two standard deviationsaway from what may be regarded as the median for subjects of similarage, weight and gender). The reference data may have been collectedsolely from a single person over time (e.g., a longitudinal record ofearlier-sensed physiologic sounds from that person), or from acollection of individuals—commonly a grouping that is demographicallysimilar (e.g., 40-50 year old healthy males weighing between 170 and 190pounds).

One particular example concerns detection of bruits in a major artery.This can be predictive of a stroke. The user may periodically place aphone or other audio sensor over the femoral (or carotid) artery. Fiveor ten seconds of collected sounds can be converted into features, asdescribed above, and these features can be compared with featuresderived from known bruits sounds. If the comparison indicates similaritybeyond a threshold degree (e.g., a feature distance below a thresholdvalue), the user can be advised to have their arteries checked forocclusion.

Patent publications 20070265508, 20110021939, 20110196820 and20130261484 detail a variety of other ways that biometric signals can beprocessed, e.g., for comparison.

As noted, speech and other vocalizations also have diagnostic value.Pitch, timbre, rhythm, pace, and volume (and variation in same), aresome of the vocalization attributes that can be monitored. Changes fromhistorical norms are sometimes symptoms or precursors of differentconditions, such as depression, stroke, alcohol poisoning, respiratoryillness, etc.

Depression, for example, is often accompanied by slower and quieterspeech, with reduced variation in pitch. Respiratory illness can bediscerned from lower-pitched speech, with rougher/coarser timbre (e.g.,due to swollen vocal cords).

Coughs may be characterized by characteristics such as frequency andtype (e.g., a dry cough—staccato in nature, with a sharp onset, shortduration and dominant high frequency components; and a wetcough—commonly the opposite). A user's cough may be matched toprototypical coughs (the user's own, or others) on a “fingerprint”basis, such as employing a collection of the features detailed in Table1, above. A user's current cough may be matched to a previous episode ofuser coughing, on a prior date. Recalling other physiologic informationfrom around that prior date may presage upcoming symptoms, e.g.,elevated body temperature, runny nose, etc.

Finer classification of coughs can be achieved with a sufficiently largecollection of reference data. A wet cough originating from irritation ofthe upper airway (e.g., mucus in the large bronchi) exhibits one set offeatures, whereas a cough originating from the lower airway (e.g.,pneumonia in the lungs) exhibits a different set of features. A distancemeasure can be employed to assess whether the set of featurescharacterizing a user's cough are closer to the former or the latter;the relative distances provide a confidence metric. More sophisticatedclassifier arrangements can also be employed.

(The audio characteristics of coughs—most especially pitch—depend, inpart, on dimensions of the acoustical bodies involved. Thus, inselecting reference data for comparison, it is desirable to pick soundscaptured from persons of the same gender, and about the same age,weight, height and—if known—lung capacity. If such a closely-matched setof reference data is not available, the feature matching operation canrely less on absolute pitch-related features, and more on otherfeatures.)

An alternative implementation to explicit distance measures is providedby classical machine learning methods. In this case, a database iscollected, consisting of appropriate audio captures, together with tagdata indicating presence of the specific types of coughs to beidentified. The audio is then processed to provide a rich set offeatures, which form the input to a machine learning paradigm (e.g. aSupport Vector Machine (SVM), or an Artificial Neural Network (ANN).Additionally, the database should include metadata to indicate gender,age, height, weight, etc. This metadata is then provided as additionalfeatures to be learned. Thus, the learning process automatically takesthese variables into account (to the extent to which the database isvaried enough to span this metadata space). In the case of an ANN, asingle network output may be provided for each cough type to berecognized; a specific cough type may then be identified by choosing thecough type with the strongest response over a predetermined threshold(assuming exclusivity among cough types). In the case of an SVM, coughsmay be classified into many (>2) classes, based on the design of severalbinary classifiers. Two popular methods are “one vs. rest” and “one vs.one.”

Medical practice, to date, has tended to classify coughs only in grosssenses (e.g., wet/dry). With the present technology, and with widespreadcollection of coughing sounds, more nuanced distinctions may be found incoughs—allowing, e.g., the causation of a particular cough to be moreaccurately determined. Thus, the present technology can help revealpreviously undiscovered knowledge.

Likewise, analysis of a large corpus of audio physiologic data (andderivatives), and correlation with diagnostic ground truth (e.g.,evaluations from physicians), can find small clues that reliablyindicate big problems. For example, it may be discovered that aconsequence of carbon monoxide poisoning is a particular change in thehigh frequency component of inhalation sounds sensed from the lowerlungs. Similarly, it may be discovered that certain types of strokes arecommonly foreshadowed by a change in the pronunciation of diphthongs(e.g., as in “oil”). When user audio and diagnostic information becomesavailable for analysis in a Big Data sense, a universe of suchstatistically-reliable correlations are likely to be found. Thereafter,a user's portable device can monitor sensed sounds (and other sensedphysiologic data) for such clues (e.g., comparing current sound signalswith historical data), and alert the user to maladies—both present andupcoming—that the user may not recognize.

Collections of reference audio data are not yet as readily available as,e.g., reference collections of mole imagery. However, such referenceinformation can be crowd-sourced in the same manners as for imagery.Since audio information is straight-forward to collect in continuousfashion, the smartphone (or other device) of each user can serve as acollection agent, and forward large amounts of audio data (or featuresderived from the audio data) to a cloud repository in a relatively shorttime.

Ground truth diagnostic data concerning such audio is harder to compile.Patients commonly don't consult physicians concerning unfamiliar coughs(as they might for unfamiliar moles); only if a person's respiratorysymptoms become severe is a physician usually involved. Moreover,pulmonary, bowel, and other audio signals escape the user's normalattention, so irregularities in those signals normally don't promptprofessional consultations.

Still further, physicians do not normally record patients' physiologicsounds—unlike the practice of photographing patients' skin conditions.Rather, a physician typically just listens to a cough, or listens tosounds heard through a stethoscope, and offers a diagnosis based on whatwas just heard. However, with the proliferation of electronic medicalrecords, and the movement towards data driven medical protocols andauditable outcomes, the digital collection and archiving of sensedpatient data—including coughs, stethoscope sounds, and all manner ofother physiologic signals (e.g., anatomic imaging by variousradiological techniques)—is expected to become more widely practiced.Thus, electronic medical records may soon provide both captured data andassociated diagnostic information, which can serve as ground truth inconnection with the present technology, eventually leading to thereliable statistical knowledge that comes with Big Data.

(In other arrangements, the physician's medical records are notemployed. Instead, a user may enter data in a personal life-log,recounting a visit to the doctor, together with the doctor's diagnosis.Such information is then associated with audio that is life-logged fromthe user—before and/or after the doctor visit. A small corpus oftraining audio, and associated diagnostic conclusions, are alsoavailable from medical schools, where they are used in training newphysicians.)

While the focus of this discussion has been on audio, it will berecognized that the same principles can be applied to collection andprocessing of other physiologic information. Without limitation, thisincludes blood pressure and blood sugar, electric signals from theheart, brain and nervous system, etc. A range of different sensors canbe employed, including electric, vibration, optical, stress, strain,chemical, etc.

A variety of multi-sensor, phone-based medical devices are known fromthe prior art, e.g., as shown in patent publications 20050124375,20090030286, 20100056880, 20120059271 and WO2013156999. Scanadu haspublicized its Scout and Scanaflo offerings, which can sensetemperature, blood pressure, heart rate, oximetry, ECG, heart ratevariability, stress, and urine chemistry—many using optical techniques.(Patent publications WO2013066642, WO2013116316, WO2013116253, andWO2014025415 detail certain of the Scanadu technologies.) Azoi, Inc.,similarly has publicized its upcoming Wello product, in which sensorsare integrated in a smartphone case, and communicate by Bluetooth to ahealth tracker app on the phone. The app logs heart rate, bloodpressure, blood oxygen, respiration, heart rate variability (as anindicator of stress), ECG, temperature, and lung function (with anaccessory spirometer).

Automated cough detection and rudimentary signal analysis is the subjectof Larson, et al, Accurate and Privacy Preserving Cough Sensing Using aLow-Cost Microphone, Proc. of the 13^(th) Int'l Conf on UbiquitousComputing, ACM, 2011, and Birring, et al, The Leicester Cough Monitor:Preliminary Validation of an Automated Cough Detection System in ChronicCough, European Respiratory Journal 31.5, pp. 1013-1018 (2008).

Many embodiments of the present technology advantageously considermultiple physiological signals together, for their diagnostic relevance(e.g., an ensemble of plural signals that co-occur in a relevant manner)or for signal processing purposes. For example, in processing signalshaving a repetitive structure (e.g., many respiratory and circulatorysignals), the signals from different repeating waveform periods can beaveraged, combined, correlated, compared, filtered or windowed. Whilethe periodicity can be inferred from, e.g., the audio itself, it cansometimes be independently determined. For example, for blood flowsounds, the periodicity can be determined by reference to electricsignals sensed from the heart (e.g., the QRS complex, or the T wave ofan EKG signal). Thus, timing derived from electric signals can beemployed in processing acoustic signals.

It will be understood that sometimes the body can be stimulated at onelocation with an audio signal or pressure waveform, and the signal canbe sensed at another location, to discern information about theintervening transmission medium. Fluid behind the tympanic membrane canbe sensed in this fashion. Dehydration can also be so-indicated, basedon the degree to which the skin is stretched or loose. Enlargement ofthe liver and constipation can also be discerned in this way, bydetection of a solid (dense) mass under the skin.

Percussion analysis is used by physicians in clinical diagnosis for theabdomen and thorax. The resonance properties of the acoustic waveformresulting from the acoustic stimulation can be examined to classify thesounds as normal, hyper-resonant, impaired resonant or dull. Formantanalysis of the captured waveforms yields information about acousticresonance. Information about associated symptoms (e.g., pain) can beused in conjunction with the captured acoustic signals for automatedanalysis.

While “sounds” and “audio” commonly refer to human-perceptible stimuli,these terms are used herein to include stimuli that may be below, and/orabove, the frequency range of human hearing. MEMS microphones used inmany smartphones, and many of the speakers, are operable well outside ofthe human hearing range. Thus, some embodiments of the technology sensedata, e.g., in the ultrasonic range. (Stimulus can also be provided inthis range.) The ability of the user device to sense sounds outside therange of human hearing provides capabilities beyond those of unassistedphysicians.

For example, with an ultrasonic-capable microphone and speaker, asmartphone may serve as a rudimentary echocardiogram device—stimulatinga portion of the body with ultrasonic audio, sensing the phase (andamplitude) of the returned signals, and presenting resultant informationon the display screen. By such arrangement, a variety of at hometesting/monitoring can be conducted, including detection of certaindilated aorta conditions.

In classifying sensed physiologic data, hidden Markov models, artificialneural networks, and deep neural networks can be employed, borrowingtechniques known from the field of pattern recognition. (Hidden Markovmodels are also known in analysis of animal vocalizations; see Ren etal, Framework for Bioacoustic Vocalization Analysis Using Hidden MarkovModels, Algorithms 2, No. 4, pp. 1410-1428, 2009.) Classification trees,support vector machines, and other discriminatory classificationtechniques can also be employed. Such classifiers use feature data fromphysiological acoustic data and known diagnoses as training sets.

The ensemble of parameters (or features) outlined above can also be usedin unsupervised learning methods to learn complex, non-linear models ofmany-dimensional underlying data. Examples of such techniques includedeep belief networks and sparse coding. (See, e.g., Raina, et al,Large-Scale Deep Unsupervised Learning Using Graphics Processors, Int'lConf. on Machine Learning, Vol. 9, 2009.) These techniques are suitablefor high-dimensional input data and can enable inference of latentvariables or conditions. Such deep learning approaches can unearth newpatterns or diagnostic tools using large numbers (even millions) ofcollected samples of various physiological acoustic data. A valuableinput to these techniques is the change of physiological acoustic dataover time. Such data can be obtained by sensing the physiologicalphenomenon (e.g., heartbeats, coughs, murmurs, etc.) at different timesover multiple days (or longer intervals).

Note that acoustic features can be combined with other availablediagnostic information (pulse rate, temperature, blood pressure, etc.)and be provided as input to automated learning and classificationmethods (both supervised and unsupervised).

Other Arrangements

A health app according to one implementation of the present technologyemploys a wearable network of sensors to capture and log a history ofphysiologic signals, and refer related information to one or more remoteprocessors (e.g., “the cloud”) for large scale systemic analysis.

Many existing sensors can be employed in such an arrangement. One is awrist-worn activity tracker, such as the Fitbit Force, Basis, Larklife,Jawbone UP, and Nike Fuelband products. As is familiar, various sensorsin these devices sense heart rate, temperature, perspiration (commonlybased on skin conductivity), and movement (e.g., based on accelerometer,magnetometer, or gyroscope sensor data). From these data, others can bederived, including calorie consumption and sleep stage.

Another type of sensor is a belt or strap, worn across the chest, waistor belly (typically horizontally, but alternatively vertically ordiagonally), which can monitor these same parameters, as well asrespiration, dimension (e.g., chest/waist/belly circumference), and bodysounds. (Obesity seems better gauged by waist- or belly-to-height ratio,rather than weight-to-height ratio.) A band on the upper arm (e.g.,across the bicep) or leg/thigh can also be employed, as can sensorsdeployed around the neck (e.g., in a necklace form) or finger (e.g., inring form). Many such sensors can be concealed under clothing

Blood pressure, blood chemistry, skin imagery data, EKG, EEG, etc., canalso be sensed. As noted, some of the sensors can be integrated intoworn clothing.

Some sensors can be responsive—in part—to stimulus introduced into thebody, such as a small electrical current, audio, vibration, etc. Forexample, electrical conductivity between two points on the body dependson the amount of intervening fat, muscle and water, as well as the skincontact resistance (which varies with perspiration). Some systems formeasuring body composition involve a user standing on two electricalpads, to sense electrical resistance; less resistance indicates lessbody fat, and a lower body fat percentage.

Audio, vibration, alternating current, and radio beam-forming arrays canused—either of emitters (e.g., piezoelectric actuators) or receptors(e.g., MEMS microphones), and employed in conjunction with one or morecomplementary receptors/emitters on another part of the body, to probeand localize characteristics (e.g., density, electrical conductivity,etc.) of the intermediate body mass, using phased array/syntheticaperture techniques known from other disciplines. (1D arrays can beused, or 2D; the sensor spacing can be spatially regular or stochastic.)By such methods, for example, the origin of a particular respiratorysound can be localized to the upper or lower lung, or to the trachea.Similarly with a murmur or other blood flow sound. (Some murmurs areabnormal based on the location where they originate, or their timingrelative to heart phase actions.)

Other non-worn sensors can also be employed, such as a weigh scale,camera, microphone, skin fold thickness calipers, sphygmomanometer,blood sugar sensor, etc. Some such sensors can be built into the user'shousehold or office environment. For example, a weigh scale may be builtinto the floor in front of a bathroom sink, and a camera may bepositioned to view the user's face when the user looks into a bathroommirror.

Still other sensors may be applied to the body, e.g., adhesively orotherwise, for short intervals, as circumstances dictate. These includeEEG and EKG electrodes, piezoelectric emitters/receptors, etc. Suchlocalized sensors may be employed to track conditions at sites ofparticular concern, e.g., bruising or other wound, cancer site, etc. Insome instances, sensors may be implanted.

All such sensing apparatus are desirably wirelessly linked, e.g., toconvey data to a user's smartphone, to each other, or to a monitoringservice.

When a user of an exemplary sensor network participates in a monitoringservice, the logged parameters—or derivative information basedthereon—are eventually sent to the cloud. The remote service monitorsthis data—noting and establishing time-of-day and day-of-week baselines,for different activity scenarios (e.g., sleep, office work, walking,strenuous exercising, etc., as classified based on characteristiccollections of sensor data). These baselines can also be associated withdifferent geographic locations, e.g., as determined by GPS or WiFi, orotherwise.

Such a monitoring service may report to the user whenever the senseddata significantly deviates from expected norms. If a person's REM sleeppulse is normally between 56 and 60 beats per second, and one nightthere is an episode in which the pulse varies from this range by athreshold amount (e.g., more than 5%, 15%, 30%, or 75%), then a messagemay be dispatched to the user (e.g., by email, text, or otherwise)noting the incident. Possible causes for the disturbance may also becommunicated to the user. These causation hypotheses can be proforma—based on textbook understandings of the noted phenomenon (e.g.,caffeine before bed) discerned from stored rule data, or they can betailored to the user—such as taking into account other user- orambient-sensor information that might be correlated (e.g., irregularrespiration, suggesting sleep apnea or the like; or an unusually warmroom—as indicated by temperature data logged by a smartphone sensor ascontrasted with historical norms—leading to increased blood flow forconvective body cooling). Such information is also logged in ahistorical data store, and may also be sent for e-charting to the user'sphysician.

Part of a health and wellness protocol may involve the user speaking aparticular phrase every day to an associated microphone sensor (e.g.,“Good morning fitband”), for collection of baseline voice information.In some instances, an interactive dialog may ensue, with the system(e.g., the user's smartphone) consulting the user's available biometricsignals and prompting—if appropriate—“You don't sound well” (indisplayed or spoken text) and asking some follow-up questions, to helpdetermine whether the user's malady is something that needs medicalattention. Through such probing (which may proceed verbally, usingspeech recognition technology), the user may reveal that she wasrefinishing furniture yesterday evening with a powerful solvent, or wasin a venue with lots of second hand smoke. Based on the collected sensorand verbal information, the system can consult stored knowledge baseinformation in advising whether the user should consult medicalpersonnel.

Another application of the present technology is in sleep analysis.Commercial sleep studies often involve instrumenting the patient withseveral different sensors, such as a band around the chest to senserespiratory effort; an O2 sensor on a finger; a sensor under the nose todetect smoothness of nasal flow; an accelerometer of the like to sensethe vibration that commonly accompanies snoring, etc. All suchinformation is sent to a data logging device. The collected informationis eventually sent to a professional for analysis.

While such arrangement may be the gold standard for sleep analysis, muchuseful information can be derived from a much simpler setup, such as oneemploying sensors resident in a smartphone phone, and—ifdesired—accessory sensors that are wirelessly sly coupled to thesmartphone. For example, vibration may be sensed by positioning thephone on the bed. Vibrations from the snoring will couple from the userto the bed, and then to the phone. (Vibratory amplitude will bediminished, and some dampening will occur, but useful informationnonetheless results. These factors may be reversed, approximately, bysignal processing, if desired.) Others of the parameters may be derivedfrom analysis of captured audio. Again, human experience does not givethe insight to discern the small clues evidencing respiratory effort,smoothness of nasal flow, etc. that are manifest in the sounds ofsnoring. But analysis of Big Data archives of such sounds—in conjunctionwith ground truth (e.g., captured from gold standard setups like the onejust reviewed)—allows such clues to be sensed and properly interpreted.

Another application of the technology involves analysis of clicking andother noises from user joints (is that degenerative arthritis, or simplecrepitus?). Still another involves stomach and bowel sounds (e.g., arethose the little bowel tones associated with healthy peristalsis, orhigh tinkling sounds that may signal an obstruction?). Yet anotherinvolves sounds accompanying urine flow. Still another involves thesounds of blood turbulence when blood starts flowing following releasein pressure from a blood pressure measurement cuff. Etc., etc. Again,lay humans typically don't discern much meaningful data from such soundsand their variations; many physicians don't do much better. But ascollections of such sounds grow, and are complemented by ground truthinterpretations by physicians, meaning emerges. The significant cluesmay not be evident in the raw audio; only by processing (e.g., bycomputing one or more of the features detailed in Table 1) may tellingsignals become apparent. But as the volume of reference data goes up,the meaning that can be derived goes up commensurately.

In many of the presently-detailed methods, historical user data is oftenthe best baseline against which to judge current user data. However,statistical aggregates of large population groups also form usefulcomparison standards. For example, even without historical information,the system can advise a user that her sleeping respiration rate of 25breaths per minute is experienced by persons of similar demographicprofile only 1% of the time, and may merit professional attention.

As noted, each user's sensed information (or derived data) is desirablysent for storage in a cloud database service, which may aggregate datafrom thousands, or millions, of people. Just as fitness tracking watches(sometimes termed “activity loggers”) have social network components(e.g., posting Facebook reports of the GPS route that users run in theirdaily jogs, with speed, distance traveled, calories, and heart rate,etc.), so too can a physiologic monitoring service. Default privacysettings can anonymize the uploaded data, but users may elect differentsettings—including to share selected categories of skin images,biometric information, activity status updates, etc., with networkfriends, without anonymization.

In some arrangements, the sensed data is associated with locationinformation—both the location on the body from which it was collected,and also the geolocation at which the information was sensed (e.g.,latitude and longitude). Data can then be recalled and analyzed,filtered, presented, etc., based on such location information. Forexample, a user can compare an average of resting pulse measurementstaken at work, with such an average taken at home. Sensed informationcan also be presented in map form. For example, a user can queryarchived data to obtain a map display that identifies locations at whichtheir resting pulse exceeded 80. (A suitable map display, detailingwhere certain data was sensed by a user, is illustrated in applicant'spublished patent application 20130324161.) Again, such information andmaps can be shared via social networking services.

Historical user data serves as a statistical chronology of the user'saging process. Again, the user's physiologic signals can be compared tothose of a relevant cohort (e.g., similarly-aged people of same gender,weight, location, etc.) to reflect whether the user is aging more orless rapidly than the norm. Relative aging may be judged by relativephysiological condition.

Generally, aging is accompanied by phenomena including higher bodyweight, lower body height, higher BMI, lower lean body mass (muscle andbone mass), higher blood pressure (both systolic and diastolic), lowermaximum oxygen use (VO2 max—Volume Maximum Oxygen consumption) underexertion (e.g., on a treadmill stress exam), decreased visual acuity(e.g., presbyopia—a lessening ability to focus on close objects),decreased hearing acuity, decreased skin tightness, etc.

Such phenomena can be sensed and used to assess physiological age, ascontrasted with chronological age, in known fashion. In comparing auser's metrics with those of a relevant cohort, variance indicates theuser is either physiologically younger, or older, than peers. Thisvariance can be tracked over time. If a user is statistically judged tohave a physiological age of 50 when actually 40, and later have aphysiological age of 52 when actually 44, then the user isphysiologically older than peers, but trending in a more healthfuldirection.

In similar fashion, trends of other biometric signals over time, e.g.,compared with statistical norms, indicate the trajectory of a person'shealth.

As the collection of reference data and the collection of ground truthdiagnostic information become larger, the frequency and/or range oferrors associated with their use become smaller. At some point, such asystem serves as a statistically reliable predictor of the future. Auser may interact with a user interface of the system to propose certainchanges (e.g., in diet, exercise, or other lifestyle) that might be madeto avoid certain undesirable predicted outcomes, and the system canpredict their respective effects.

Many apps are downloaded to users' phones, and left un-used after a weekor so. In part, this is due to the hurdle of having to awaken the phonefrom sleep mode whenever use of the app is desired. Technologies arearising, however, (such as applicant's patent application 61/920,722,filed Dec. 24, 2013) in which portable user devices can maintain somefunctionality—such as audio responsiveness—even when “asleep.” (In someembodiments, this capability relies on use of an auxiliary processorthat is active all the time, and which consumes just a tiny fraction ofthe power consumed by the phone's primary processor(s).)

In accordance with another aspect of the technology, the audio and/orother physiologic signal sensing described herein is on-going even whenthe screen of the device is dark and the device is otherwise in a“sleep” mode. To reduce battery drain, the data collected in this modeis not streamed to the cloud, but rather is cached in a memory until thephone is next awakened. Similarly, the processing of the captured datato produce derivative data may be queued, e.g., waiting until thedevice's primary processor is again available. Thus, data collectionaspects of a health app may be on-going, 24 hours a day, 7 days a week.

As noted, individual privacy is a concern with such technologies. Yet,so too is the importance of establishing a pool of shared informationthat is sufficiently large to bring benefits of scale and meaningfulstatistics. Users may sometimes serve as providers of their owninformation, and consumers of others' information. In such situation,there can be a marketplace for exchange of physiologic information.Those whose consumption is out of proportion to their sharing may becalled on to pay, or offer other consideration, for their relativelyhigh consumption (or, put another way, for their relatively heightenedprivacy). Those who share more than they consume may receive cash orother consideration to reward their relative openness. The marketplacecan establish the equilibrium point between giving and taking, where noexchange of consideration is required.

(Some categories of information, e.g., carotid artery sounds, may bescarcer than others. Similarly, information from certain demographicsegments, e.g., young women in North Dakota, may be scarcer than others.The exchange marketplace can take such factors into account inestablishing pricing for information.)

In some arrangements, a person's consultation with a physician may notbe prompted by use of the present technology; rather, a consultationwith a physician may prompt use of the present technology.

For example, a patient may visit a physician prompted by an onset ofwheezing in their respiration. On investigation, the physician maydetect a murmur (which may be associated with the wheezing). Thephysician may instruct the patient to capture, twice-daily, sounds ofthe murmur by positioning the phone in a certain location on the chest.Such in-home data collection can then inform judgments by the physicianabout further care.

Similarly, a physician who diagnoses possible depression in a patientcan use voice data from at-home data collection to determine whether thecondition is trending better or worse. (The physician needn't review theactual recorded voice. Parameters expressing volume, pitch, andvariations in same, can be derived from the patient's sensed voice, andsuccinctly reported for physician review.)

A hundred years ago, physicians didn't have labs that could analyze abit of body fluid to produce a page full of quantitative data; theyrelied on their eyes, and ears, and their fingers. Lab data gives greatinsight into patient conditions, but something is lost when lab analysisreplaces physical exam—as often now seems to be the case. Applicationsof the present technology redress this shortcoming, employing digitalsensors that are more acute and accurate than any physician's senses.Coupled with insights from Big Data, digital sensor-assisted physicalexam may again emerge as the primary tool of diagnostic medicine.

(As noted elsewhere—but it bears repeating, it is poor practice to makean automated diagnosis based on sensed data and statistics. But suchinformation allows systems according to the present technology toconclude that something seems different or amiss, so that such fact canbe revealed and acted on by a professional.)

Review

A very few of the novel arrangements detailed herein are reviewed inrough summary fashion, below.

One aspect of the present technology involves collecting physiologicinformation in a data structure, where the collected informationcorresponds to physiologic sensor data gathered by pluralnon-professional users. In the data structure is also collectedprofessional evaluation information corresponding to at least some ofthe physiologic information. Thereafter, query information is received,corresponding to physiologic sensor data gathered by a non-professionaluser. The data structure is consulted in determining result information,and at least some of this result information is communicated to thenon-professional user. This result information depends on correlationbetween the query information, the collected physiology information, andthe professional evaluation information.

Another arrangement employs a portable device that is moved to pluraldifferent viewpoint positions relative to a skin location. At a first ofthese viewpoint positions, the skin location is illuminated with lightof a first spectrum from the portable device. While so-illuminated,imagery of the skin location is captured using a camera in the portabledevice. At a second of these viewpoint positions, the skin location isilluminated with light of a second spectrum from the portable device.Again, while so-illuminated, imagery of the skin location is capturedusing the portable device. Thus, the skin location is imaged by theportable device from plural different viewpoint positions and withplural different illumination spectra. The captured imagery may then beprocessed and, based on such processing, the user may be advised whetherto seek a professional evaluation of the skin location.

In a related method, the skin location is illuminated with light of afirst spectrum from a first region of a display of the portable device,while imagery of the skin location is captured. Similarly, the skinlocation is illuminated with light of a second spectrum from a secondregion of the portable device display. Again, imagery of the skinlocation is captured while illuminated with this second spectrum oflight. By such arrangement, imagery of the skin location is capturedwith plural different illumination angles and plural differentillumination spectra.

Still another related method involves presenting a first illuminationfrom a portable device display screen to a subject at a first time, andcapturing a first image of the subject when it is so-illuminated.Likewise, second illumination is presented from the display screen tothe subject at a second time, and a second image of the subject,so-illuminated, is captured. In such arrangement, the illumination doesnot comprise a viewfinder rendering of captured imagery.

Another aspect of the technology involves capturing image informationfrom a reference subject, using a camera, and processing the capturedimage information to yield reference color information. Imagery is alsocaptured with the camera, depicting a skin rash or lesion. This latterimagery is then color-corrected by use of the reference colorinformation. (The reference subject can comprise, e.g., blood or abanknote.)

Still another aspect of the technology involves receiving imagerycomprising plural frames depicting a skin rash or lesion. A processingoperation is invoked on the plural frames, to yield an enhanced stillimage (e.g., (a) a super-resolution image, (b) a noise-reduced image,(c) a multi-spectral image, (d) an ambient light-compensated image, or(e) a 3D image). The enhanced still image (or data derived therefrom) issubmitted to a database for similarity-matching with reference imagesdepicting skin rashes or lesions, or data derived from such referenceimages. Such database includes reference data corresponding to enhancedstill images that themselves have been processed from plural-frameimagery.

Yet another arrangement according to the present technology involvesobtaining plural sets of professional data, where each set includes skinimage data and patient profile data. This patient profile data includesboth opinion information provided by a medical professional, and factualinformation. From the skin image data, first feature information isextracted (e.g., using a hardware processor configured to perform suchact). Plural sets of lay data are also obtained, where this lay dataincludes skin image data and patient profile data (e.g., fact data), butlacks information provided by a medical professional. From the skinimage data in the lay data, second feature information is extracted. Thefirst and second extracted feature information is then made available asreference feature information for similarity-matching with featureinformation extracted from query skin image data.

Still another aspect of the technology involves receiving data,including skin image data and associated metadata, from a party.Similarities between the received data and reference data aredetermined. This reference data includes multiple sets of data, eachincluding skin image data and associated metadata. Included among thereference data are sets of data that have been professionally curated,and also sets of data that have not been professionally curated.

A further method involves receiving a first set of information from afirst submitter, where the first set of information includes imagerydepicting a part of a first subject's body that evidences a symptom of afirst pathological condition, and also includes drug profile dataindicating drugs taken by the first subject. A second set of informationis received from a second submitter. This second set of informationincludes imagery depicting a part of a second subject's body thatevidences a symptom of a second pathological condition, and alsoincludes drug profile data indicating drugs taken by the second subject.Such sets of information are received from 3d through Nth submitters.Information is then received corresponding to a query image submitted bya user. One or more image parameters are computed from this query image,and compared for correspondence against such parameters computed for theimagery received from the first through Nth submitters. Information isthen sent to the user, identifying one or more drugs that is correlatedwith skin imagery (e.g., skin symptoms) having an appearance like thatin the query image.

Yet another method involves receiving a first set of information from afirst submitter, where this first set of information includes imagerydepicting a part of a first subject's body that evidences a symptom of afirst pathological condition, and also includes a diagnosis of the firstpathological condition. Such information is likewise received from asecond submitter, including imagery depicting part of a second subject'sbody that evidences a symptom of a second pathological condition, andalso includes a diagnosis of the second pathological condition. This isrepeated for third through Nth sets of information, received from thirdthrough Nth submitters. Information is then received corresponding to aquery image submitted by a user. One or more parameters are computedfrom the query image. A search is conducted to identify imagery receivedfrom the first through Nth submitters having correspondence with thecomputed image parameter(s). Information (e.g., information aboutcandidate diagnoses, or about diagnoses that are inconsistent with theavailable information) are then sent to the user, based on suchsearching of imagery.

Still another method involves obtaining first imagery depicting a partof a mammalian body that evidences a symptom of a possible pathologicalcondition. This imagery is processed to derive one or more imageparameter(s). A data structure is searched for reference information,based on the derived image parameter(s). Result information isdetermined as a consequence of such search. This reference informationcomprises information identifying one or more particular pathologicalconditions that is not the pathological condition evidenced by thedepicted part of the body. At least some of this result information iscommunication to a user.

A further aspect of the technology involves sensing whether a mobiledevice is in a static, viewing pose. When it is not, frames of imagerydepicting a user are collected. When it is in the static viewing pose,information is presented for user review. This presented information isbased, at least in part, on the collected frames of imagery. In sucharrangement, the mobile device automatically switches between datacollection and data presentation modes (e.g., based on pose and/ormotion).

A further aspect of the technology involves capturing audio sounds froma user's body, using a portable device held by the user. Plural featuresare derived from the captured audio; these features comprise fingerprintinformation corresponding to the captured sounds. This fingerprintinformation is provided to a knowledge base (data structure), whichcontains reference fingerprint data and associated metadata. Metadataassociated with one or more of the reference fingerprint data in theknowledge base is then received back by the device. Information based onthis received metadata (e.g., physiologic- or health-relatedinformation) is presented to the user.

Still another aspect of the present technology concerns an imaging boothdefined by one or more sidewalls. A plurality of cameras are disposedalong the one or more side walls, directed toward an interior of thebooth, and these cameras are connected to an image processor. The boothalso includes a plurality of light sources directed toward the interiorof the booth, coupled to driving electronics. Plural of these lightsources is each spectrally tuned to a different wavelength, and thedriving electronics are adapted so that different of the lightingelements illuminate at different times, causing the cameras to captureimagery under plural different spectral lighting conditions. The imageprocessor is adapted to produce spectricity measurements based onimagery from the cameras.

CONCLUDING REMARKS

Having described and illustrated the principles of the inventive workwith reference to illustrative examples, it will be recognized that thetechnology is not so limited.

For example, medical diagnosis often relies heavily on patient history.Such history information can be extracted from medical records, and usedin assessing the possible diagnostic relevance of different physiologicsignals, or combinations of signals.

To illustrate, crackles or rales in the lungs, for most people, mayindicate pneumonia—even if accompanied by a gain in weight. However, ifthese symptoms appear in a person known to be suffering from chronicheart failure, then the lung sounds take on a new significance—one thatrequires a prompt visit to the doctor.

Likewise, DNA testing is becoming more commonplace. A physiologic signal(e.g., crackles or rales) may take on new meaning when interpreted inthe context of particular DNA findings (which may indicate, e.g., asusceptibility to particular viral illnesses).

Although the focus of this disclosure has been on capture of imagery,audio, and more familiar physiologic data, mention should also be madeof haptics. Haptic technology allows data to be acquired concerningtactile information—such as the firmness, tautness, elasticity orresilience of a body part. (See, e.g., Murayama, et al, Development of aNew Instrument for Examination of Stiffness in the Breast Using HapticSensor Technology, Sensors and Actuators A: Physical 143.2, pp. 430-438,2008.) In addition to haptic sensors, haptic actuators can also beemployed—applying physical forces to the body in controlled direction,strength, and temporal pattern, so that measurement data responsive tosuch stimulus can be sensed. Again, by collection of a body of hapticinformation, and correlation with physicians' diagnostic ground truth,certain haptic data (or its derivatives) can be discovered to havediagnostic significance.

While the Face-Chek mode noted above employed rainbow mode illumination,it will be recognized that the Face-Chek mode can also use otherlighting arrangements, e.g., simple ambient light.

Certain of the described arrangements may capture imagery in which thebody of the camera device (e.g., smartphone) casts a visible shadow.Arrangements for detailing with such shadows are detailed in applicant'sU.S. Pat. No. 8,488,900.

While one of the above-detailed dermatological embodiments employed abrute force, exhaustive search through a knowledge base to assesssimilarities with reference image data, more sophisticated methods cannaturally be employed.

One is to provide indices to the database, sorted by differentparameters. Thus in the case of a simple scalar parameter that rangesfrom 0-100, if the query image has a parameter of 37, then a binary orother optimized search can be conducted in the index to quickly identifyreference images with similar parameter values. Reference images withremote values needn't be considered for this parameter. (Most of thedetailed image derivatives are vector parameters, comprising multiplecomponents. Similar database optimization methods can be applied.)

Still further, known machine learning techniques can be applied to thereference data to discern which image derivatives are most useful asdiagnostic discriminants of different conditions. When a query image isreceived, it can be tested for these discriminant parameters to morequickly perform a Bayesian evaluation of different candidate diagnosishypotheses.

Bag-of-features techniques (sometimes termed “bag of words” techniques)can also be mployed to ease, somewhat, the image matching operation (butsuch techniques “cheat” by resort to data quantization that may—in someinstances—bias the results).

Other pattern recognition techniques developed for automated molediagnosis can likewise be adapted to identifying database images thatare similar to a query image.

Certain embodiments of the present technology can employ existing onlinecatalogs of imagery depicting different dermatological symptoms andassociated diagnoses. Examples include DermAtlas, atwww<dot>dermatlas<dot>org—a crowd-sourced effort managed by physiciansat Johns Hopkins University) and DermNet NZ atwww<dot>dermnetnz<dot>org—a similar effort by the New ZealandDermatological Society. Similarly-named to the latter is Dermnet, a skindisease atlas organized by a physician in New Hampshire, based onsubmittals from various academic institutions, www<dot>dermnet<dot>com.Also related is the website Differential Diagnosis in Dermatology,www<dot>dderm<dot>blogspot<dot>com, and the web site of theInternational Dermoscopy Society, www<dot>dermoscopy-ids<dot>org.

Sometimes patient privacy rights (e.g., HIPAA) pose an impediment tocollection of imagery, even for anonymous, crowd-source applications. Assuggested above, one approach to collection of crowd-sourced imagery isto offer financial incentives to patients to induce them to share theirmole imagery or other physiologic information.

While one of the detailed arrangements presented a ranked listing ofpossible pathologies to consider (or to rule out), other embodiments canpresent information otherwise, e.g., with other representations ofconfidence. Histograms, heat maps, and phylogenetic diagrams areexamples.

In some embodiments, the technology serves as an advisor to a medicalprofessional—offering suggested diagnoses, or further testing, toconsider. The offered advice may be tailored in accordance with wishesof the professional, e.g., expressed in stored profile datacorresponding to that professional. For example, one practitioner mayexpress a conservative medical philosophy, in which case such anadvisory service may offer only observations/suggestions in which thereis a high degree of confidence. Conversely, another practitioner may bemore open to novel theories and approaches, in which case the system mayalso present candidate diagnoses (and further testing suggestions) thatis more speculative.

Similarly, some practitioners may specify diagnostic criteria that theytend to weigh more (or less) heavily in reaching particular conclusions.In dermatology, for instance, one physician may regard chromaticdiversity across a mole as particularly relevant to a diagnosis ofmalignant melanoma. Another physician may not hold such criterion inhigh regard, but may find scalloped edge contours to be highly probativefor such a diagnosis. Again, stored profile data for the differentprofessionals can indicate such preferences, and tailor system responseaccordingly. (In some arrangements, such preferences are not expresslyspecified as profile information by the physician, but rather arededuced through analysis of electronic medical records detailingprevious diagnoses, and the clinical data on which they were based.)

In processing imagery, e.g., of moles, known photogrammetry techniquesare desirably employed to mitigate for pose distortion and cameraoptics, to yield an orthorectified image (aka an orthoimage). So doingenhances the statistical matching of user-submitted skin imagery withpreviously-submitted imagery.

While the skin conditions discussed above are organic in nature, thesame principles can be applied to skin conditions that result fromtrauma, including bug bites and wounds. A user who returns from avacation with a painful leg bite may wonder: Is it a spider bite? Aflea? A bed bug? A scrape that doesn't heal well, and turns red andangry, may be another cause for concern: Is that a staph infection? Asin the cases detailed above, a suitably-large knowledge base can revealanswers.

Reference was made to sensed information, and other information derivedtherefrom. This latter information is variously referred to in thisspecification, e.g., as features, parameters, fingerprints, deriveddata, etc. It should be understood that these latter terms areinterchangeable.

Another form of metadata that may be associated with sensed user data(e.g., skin images) is information indicating treatments the user hastried, and their assessment of success (e.g., on a 0-10 scale). In theaggregate, such data may reveal effective treatments for different typesof rashes, acne, etc.

Skin also serves as a barometer of other conditions, including emotion.Each emotion activates a different collection of bodily systems,triggering a variety of bodily responses, e.g., increased blood flow(vasocongestion) to different regions, sometimes in distinctive patternsthat can be sensed to infer emotion. (See, e.g., Nummenmaa, et al,Bodily Maps of Emotions, Proceedings of the National Academy of Sciencesof the United States of America, 111, pp. 646-651, 2014.) Just as skinconductivity is used in some lie detectors, so too may skin imagery.

In some systems, sensed physiologic information and its derivatives arerepresented in the form of Linked Data—both for individual and aggregatedata, and both for storage and for sharing—in order to facilitatesemantic reasoning with such information. (The artisan is presumed to befamiliar with Linked Data and related constructs, e.g., as popularizedby Tim Berners-Lee, as standardized by the World Wide Web Consortium,and as discussed in the assignee's patent publications including20110098056 and 20120154633.)

In other embodiments, imagery, image derivatives, and metadatainformation are stored in accordance with the DICOM standards formedical image records (see, e.g.,www<dot>dclunie<dot>com/dicom-status/status<dot>html).

Certain embodiments of the technology recognize the user's forearm orother body member (e.g., by classification methods), and use thisinformation in later processing (e.g., in assessing scale of skinfeatures). In some such arrangements, analysis is applied to videoinformation captured while the user is moving the smartphone camera intoposition to capture skin imagery. Such “flyover” video is commonly oflower resolution, and may suffer from some blurring, but is adequate forbody member-recognition purposes. If, e.g., a hand is recognized fromone or more frames of such video, and the smartphone is thereafter moved(as sensed by accelerometers and/or gyroscopes) in a manner consistentwith that hand being the ultimate target for imaging (e.g., thesmartphone is moved in a direction perpendicular to the plane of thephone screen—moving it closer to the hand), then the subject of theimage is known to be the hand, even if the captured diagnostic imageitself is a close-up from which the body location cannot be deduced.

Many of the techniques described in connection with humans can also beapplied to animals. Unusual skin conditions can be expanded to animalhide, fur and feathers (although false positives and hidden conditionsmay be more likely with complex skin coverings). Vets often face a moredifficult challenge than physicians, since animals cannot describesymptoms that might aid in diagnosis, making the notion of providing acandidate list of maladies and being able to quickly test for additionalsymptoms even more valuable. Pet owners often need to decide whethersymptoms warrant a visit to a vet and whether particular visiblesymptoms can be explained by recent known activities of that pet.Furthermore, livestock owners face the challenge of outbreaks ofcontagious diseases and need to inspect their animals often to catchsuch diseases as early as possible. Pet and livestock owners can benefitgreatly from the present technology for examining and diagnosingconditions.

For livestock owners, an automated early warning system can be set inplace where livestock passing through gates or paddocks are routinelyexamined for unusual skin variations that suggest closer examination isneeded. Livestock are often outfitted with RF tags for identification,allowing such a monitoring system to compare individual livestock overtime to rule out health conditions that have already been addressed, andto note new, emerging conditions. Wildlife managers can also benefit bysetting up imaging systems on commonly traversed paths that aretriggered by passing animals. Again, early detection and identificationof contagious conditions or dangerous pests is key to maintaininghealthy populations.

Another diagnostically useful feature is temporal observation of bloodflow through the area of a skin condition. Subtle color changes due tolocal blood pressure modulated by heartbeats can be used to distinguishbetween, or assess the severity of, some skin conditions. One method ofobserving these subtle color changes is given in the Wu paper citedbelow (“Eulerian Video Magnification for Revealing Subtle Changes in theWorld”), where small differences are magnified through spatio-temporalsignal processing. Elasticity of a region can be measured by applyingpressure (by machine or by touch) in such a way as to bend the skin. Bycomparing various points on the skin before and after deformation(ideally, a repeated pattern of deformation to allow for averaging), thelocal elastic properties of the skin can be included in the diagnosis.

While 3D considerations were noted above (e.g., in regard tostructure-from-motion methods), the local 3D texture of the skincondition region can also be assessed through the use of 3D imagingtechnology, including light-field and plenoptic cameras. One canconsider both the angle of illumination of the incident light withrespect to the imaging sensor as well as a lens cluster that providesdepth variations as a byproduct of the imaging method.

Image analysis in the Lab color space is often preferred to RGB-basedanalysis, since normal skin color is a relatively small region in (a,b).The value of L (luminance) depends on the concentration of melanin, theskin color dye.

Reference was made to surface topology, and methods regarding same.Accurate 3D information can also obtained from a single camera system byilluminating a region of interest of the patient with a structured lightpattern. Distortions in the structured light pattern are used todetermine the 3D structure of the region, in familiar manner. Thepattern may be projected, e.g., by a projector associated with thecamera system. (E.g., a mobile phone or headworn apparatus can include apico data projector.)

Another group of image processing techniques useful in diagnosticanalyses is Mathematical Morphology (see, e.g., the Wikipedia article ofthat name), where the topology of an image is described in terms ofspatial surface descriptions. This is used, e.g., in counting of smallcreatures/structures under a microscope. Such technology is well suitedto counting “bumps” or other structures per area in a skin lesion. Italso allows for representation by attributed relational graphs thatdescribe a detailed relationship between structures that can be comparedas graphs independent of orientation and specific configuration.

It will be recognized that the term “lesion” is used in thisspecification in a generic sense, e.g., referring to any feature of theskin, including spots, moles, rashes, nevi, etc.

While reference was made to app software on a user's smartphone asperforming certain of the detailed functionality, in other embodimentsthese functions can naturally be performed otherwise—including byoperating system software on a smartphone, by a remote server, byanother smartphone or computer device, distributed between such devices,etc.

While reference has been made to smartphones, it will be recognized thatthis technology finds utility with all manner of devices—both portableand fixed. Tablets, laptop computers, digital cameras, wrist- andhead-mounted systems and other wearable devices, servers, etc., can allmake use of the principles detailed herein. (The term “smartphone”should be construed herein to encompass all such devices, even thosethat are not telephones.)

Reference was made to “bag of features” techniques. Such methods extractlocal features from patches of an image (e.g., SIFT points), andautomatically cluster the features into N groups (e.g., 168 groups)—eachcorresponding to a prototypical local feature. A vector of occurrencecounts of each of the groups (i.e., a histogram) is then determined, andserves as a reference signature for the image, or for a sub-partthereof. To determine if a query image matches the reference image,local features are again extracted from patches of the image, andassigned to one of the earlier-defined N-groups (e.g., based on adistance measure from the corresponding prototypical local features). Avector occurrence count is again made, and checked for correlation withthe reference signature. Further information is detailed, e.g., inNowak, et al, Sampling strategies for bag-of-features imageclassification, Computer Vision-ECCV 2006, Springer Berlin Heidelberg,pp. 490-503; and Fei-Fei et al, A Bayesian Hierarchical Model forLearning Natural Scene Categories, IEEE Conference on Computer Visionand Pattern Recognition, 2005; and references cited in such papers.

Some of applicant's related work, e.g., concerning imaging, imageprocessing systems, and related smartphone apps is detailed in patentpublications 20110212717, 20110161076, 20120284012, 20120046071,20140052555, 20130329006, 20140057676, and in pending application Ser.No. 13/842,282, filed Mar. 15, 2013, Ser. No. 14/251,229, filed Apr. 11,2014, 61/861,931, filed Aug. 2, 2013, and Ser. No. 13/969,422, filedAug. 16, 2013.

Several references have been made to application Ser. No. 14/201,852. Inaddition to extensive disclosure of several multi-spectral imagingtechniques, that document teaches a variety of other arrangements thatare useful in conjunction with the present technology. These includetechniques for mitigating errors in spectricity measurements,compensation for field angle non-uniformities, various classificationmethods (including vector quantization, support vector machines, andneural network techniques), different object recognition technologies,and image comparison based on the freckle transform data, among others.

SIFT is an acronym for Scale-Invariant Feature Transform, a computervision technology pioneered by David Lowe and described in various ofhis papers including “Distinctive Image Features from Scale-InvariantKeypoints,” International Journal of Computer Vision, 60, 2 (2004), pp.91-110; and “Object Recognition from Local Scale-Invariant Features,”International Conference on Computer Vision, Corfu, Greece (September1999), pp. 1150-1157, as well as in U.S. Pat. No. 6,711,293. Additionalinformation about SIFT (and similar techniques SURF and ORB) is providedin the patent documents cited herein.

While SIFT is referenced, other robust feature points may be preferredfor skin imagery. For example, SIFT is typically performed on grey-scaleimagery; color is ignored. In contrast, feature points for skin canadvantageously employ color. An exemplary set of feature points specificto close-up skin imagery can comprise skin pores (or hair follicles).The center of mass of each such feature is determined, and the pixelcoordinates of each are then associated with the feature in a datastructure. In other arrangements, 3D features can additionally oralternatively be used. Features can also be drawn from those that arerevealed by infrared sensing, e.g., features in the dermal layer,including blood vessel minutiae. (See, e.g., Seal, et al, AutomatedThermal Face Recognition Based on Minutiae Extraction, Int. J.Computational Intelligence Studies, 2013, 2, 133-156, attached toapplication 61/872,494, and references cited therein.)

The design of smartphones and other such devices reference herein isfamiliar to the artisan. In general terms, each includes one or moreprocessors, one or more memories (e.g. RAM), storage (e.g., a disk orflash memory), a user interface (which may include, e.g., a keypad, aTFT LCD or OLED display screen, touch or other gesture sensors, a cameraor other optical sensor, a compass sensor, a 3D magnetometer, a 3-axisaccelerometer, a 3-axis gyroscope, one or more microphones, etc.,together with software instructions for providing a graphical userinterface), interconnections between these elements (e.g., buses), andan interface for communicating with other devices (which may bewireless, such as GSM, 3G, 4G, CDMA, WiFi, WiMax, Zigbee or Bluetooth,and/or wired, such as through an Ethernet local area network, a T-1internet connection, etc.).

The processes and system components detailed in this specification canbe implemented as instructions for computing devices, including generalpurpose processor instructions for a variety of programmable processors,including microprocessors (e.g., the Intel Atom, the ARM A5, and theQualcomm Snapdragon, and the nVidia Tegra 4; the latter includes a CPU,a GPU, and nVidia's Chimera computational photography architecture),graphics processing units (GPUs, such as the nVidia Tegra APX 2600, andthe Adreno 330-part of the Qualcomm Snapdragon processor), and digitalsignal processors (e.g., the Texas Instruments TMS320 and OMAP seriesdevices), etc. These instructions may be implemented as software,firmware, etc. These instructions can also be implemented in variousforms of processor circuitry, including programmable logic devices,field programmable gate arrays (e.g., the Xilinx Virtex series devices),field programmable object arrays, and application specificcircuits—including digital, analog and mixed analog/digital circuitry.Execution of the instructions can be distributed among processors and/ormade parallel across processors within a device or across a network ofdevices. Processing of signal data may also be distributed amongdifferent processor and memory devices. “Cloud” computing resources canbe used as well. References to “processors,” “modules” or “components”should be understood to refer to functionality, rather than requiring aparticular form of implementation.

Software instructions for implementing the detailed functionality can beauthored by artisans without undue experimentation from the descriptionsprovided herein, e.g., written in C, C++, Visual Basic, Java, Python,Tcl, Perl, Scheme, Ruby, etc. Smartphones and other devices according tocertain implementations of the present technology can include softwaremodules for performing the different functions and acts.

Software and hardware configuration data/instructions are commonlystored as instructions in one or more data structures conveyed bytangible media, such as magnetic or optical discs, memory cards, ROM,etc., which may be accessed across a network. Some embodiments may beimplemented as embedded systems—special purpose computer systems inwhich operating system software and application software areindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

The databases and other data structures referenced herein can bemonolithic, or can be distributed. Thus, reference data may be storedanywhere, e.g., user devices, remote device, in the cloud, dividedbetween plural locations, etc.

While the specification described certain acts as being performed by theuser device (phone) or by the central system, it will be recognized thatany processor can usually perform any function. For example, computationof image derivatives, and color correction, can be done by the user'ssmartphone or by the central system—or distributed between variousdevices. Thus, the fact that an operation is described as beingperformed by one apparatus, should be understood as exemplary and notlimiting.

In like fashion, description of data being stored on a particular deviceis also exemplary; data can be stored anywhere: local device, remotedevice, in the cloud, distributed, etc.

As indicated, the present technology can be used in connection withwearable computing systems, including headworn devices. Such devicestypically include one or more sensors (e.g., microphone(s), camera(s),accelerometers(s), etc.), and display technology by which computerinformation can be viewed by the user—either overlaid on the scene infront of the user (sometimes termed augmented reality), or blocking thatscene (sometimes termed virtual reality), or simply in the user'speripheral vision. A headworn device may further include sensors fordetecting electrical or magnetic activity from or near the face andscalp, such as EEG and EMG, and myoelectric signals—sometimes termedBrain Computer Interfaces, or BCIs. (A simple example of a BCI is theMindwave Mobile product by NeuroSky, Inc.) Exemplary wearable technologyis detailed in patent documents 7,397,607, 20100045869, 20090322671,20090244097 and 20050195128. Commercial offerings, in addition to theGoogle Glass product, include the Vuzix Smart Glasses M100, Wrap 1200AR,and Star 1200XL systems. An upcoming alternative is augmented realitycontact lenses. Such technology is detailed, e.g., in patent document20090189830 and in Parviz, Augmented Reality in a Contact Lens, IEEESpectrum, September, 2009. Some or all such devices may communicate,e.g., wirelessly, with other computing devices (carried by the user orotherwise), or they can include self-contained processing capability.Likewise, they may incorporate other features known from existing smartphones and patent documents, including electronic compass,accelerometers, gyroscopes, camera(s), projector(s), GPS, etc.

As noted, embodiments of present technology can also employ neuromorphicprocessing techniques (sometimes termed “machine learning,” “deeplearning,” or “neural network technology”). As is familiar to artisans,such techniques employ large arrays of artificial neurons—interconnectedto mimic biological synapses. These methods employ programming that isdifferent than the traditional, von Neumann, model. In particular,connections between the circuit elements are weighted according tocorrelations in data that the processor has previously learned (or beentaught).

Each artificial neuron, whether physically implemented or simulated in acomputer program, receives a plurality of inputs and produces a singleoutput which is calculated using a nonlinear activation function (suchas the hyperbolic tangent) of a weighted sum of the neuron's inputs. Theneurons within an artificial neural network (ANN) are interconnected ina topology chosen by the designer for the specific application. In onecommon topology, known as a feed-forward network, the ANN consists of anordered sequence of layers, each containing a plurality of neurons. Theneurons in the first, or input, layer have their inputs connected to theproblem data, which can consist of image or other sensor data, orprocessed versions of such data. Outputs of the first layer areconnected to the inputs of the second layer, with each first layerneuron's output normally connected to a plurality of neurons in thesecond layer. This pattern repeats, with the outputs of one layerconnected to the inputs of the next layer. The final, or output, layerproduces the ANN output. A common application of ANNs is classificationof the input signal into one of N classes (e.g., classifying a type ofmole). In this case the output layer may consist of N neurons inone-to-one correspondence with the classes to be identified.Feed-forward ANNs are commonly used, but feedback arrangements are alsopossible, where the output of one layer is connected to the same or toprevious layers.

Associated with each connection within the ANN is a weight, which isused by the input neuron in calculating the weighted sum of its inputs.The learning (or training) process is embodied in these weights, whichare not chosen directly by the ANN designer, In general, this learningprocess involves determining the set of connection weights in thenetwork that optimizes the output of the ANN is some respect. Two maintypes of learning, supervised and unsupervised, involve using a trainingalgorithm to repeatedly present input data from a training set to theANN and adjust the connection weights accordingly. In supervisedlearning, the training set includes the desired ANN outputscorresponding to each input data instance, while training sets forunsupervised learning contain only input data. In a third type oflearning, called reinforcement learning, the ANN adapts on-line as it isused in an application. Combinations of learning types can be used; infeed-forward ANNs, a popular approach is to first use unsupervisedlearning for the input and interior layers and then use supervisedlearning to train the weights in the output layer.

When a pattern of multi-dimensional data is applied to the input of atrained ANN, each neuron of the input layer processes a differentweighted sum of the input data. Correspondingly, certain neurons withinthe input layer may spike (with a high output level), while others mayremain relatively idle. This processed version of the input signalpropagates similarly through the rest of the network, with the activitylevel of internal neurons of the network dependent on the weightedactivity levels of predecessor neurons. Finally, the output neuronspresent activity levels indicative of the task the ANN was trained for,e.g. pattern recognition. Artisans will be familiar with the tradeoffsassociated with different ANN topologies, types of learning, andspecific learning algorithms, and can apply these tradeoffs to thepresent technology.

Another machine learning arrangement that is well suited for embodimentsof the present technology is support vector machines (SVMs). SVMs aredetailed, e.g., in U.S. Pat. Nos. 6,157,921, 6,714,925, 7,797,257 and8,543,519.

Additional information on such techniques is detailed in the Wikipediaarticles on “Machine Learning,” “Deep Learning,” and “Neural NetworkTechnology,” as well as in Le et al, Building High-Level Features UsingLarge Scale Unsupervised Learning, arXiv preprint arXiv:1112.6209(2011), and Coates et al, Deep Learning with COTS HPC Systems,Proceedings of the 30th International Conference on Machine Learning(ICML-13), 2013. These journal papers, and then-current versions of the“Machine Learning” and “Neural Network Technology” articles, areattached as appendices to copending patent application 61/861,931, filedAug. 2, 2013. application Ser. No. 14/201,852 also has a discussion ofmachine learning useful with the present technology.

Reference was made to statistical findings based on the reference data.The artisan is presumed to be familiar with statistics and their use.

In some of the foregoing examples, reference was made to a conditionbeing statistically unlikely or improbable. The particular thresholdused in such determinations can be set by the implementer, based on therequirements of the particular application. In some arrangements, aprobability of less than 1% may be deemed statistically unlikely. Inothers, a probability of less than 0.3%, 0.1%, 0.03% or 0.01% may berequired.

Pulse detection from wearable clothing and devices is taught, e.g., inU.S. Pat. Nos. 5,622,180, 6,104,947 and U.S. Pat. No. 7,324,841 to PolarElectro OY. Other related writings include patent documents U.S. Pat.Nos. 6,021,344, 6,606,628, 6,882,990, 7,233,693, 20020021828,20080194928, 20110301441, 2012008838, 20120308086, and WO13070895, andthe following other publications (all appended to application61/832,715):

-   Arafini, “Dermatological disease diagnosis using color-skin images,”    2012 Intl Conf on Machine Learning and Cybernetics;-   Bersha, “Spectral Imaging and Analysis of Human Skin,” Master's    Thesis, University of Eastern Finland, 2010;-   Cavalcanti, et al, “An ICA-based method for the segmentation of    pigmented skin lesions in macroscopic images, IEEE Int'l Conf on    Engineering in Medicine and Biology Society, 2011;-   Cavalcanti, et al, Macroscopic pigmented skin lesion segmentation    and its influence on lesion classification and diagnosis, Color    Medical Image Analysis. Springer Netherlands, 2013, pp. 15-39;-   Korotkov et al, “Computerized analysis of pigmented skin lesions—a    review,” Artificial Intelligence in Medicine 56, pp. 69-90 (2012);-   Parolin, et al, “Semi-automated diagnosis of melanoma through the    analysis of dermatological images,” 2010 23rd IEEE SIBGRAPI    Conference on Graphics, Patterns and Images;-   Sadeghi et al, “Detection and analysis of irregular streams in    dermoscopic images of skin lesions,” preprint, IEEE Trans. on    Medical Imaging, 2013;-   Sadeghi, et al, “Automated Detection and Analysis of Dermoscopic    Structures on Dermoscopy Images,” 22nd World Congress of    Dermatology, 2011; and-   Wu, Eulerian Video Magnification for Revealing Subtle Changes in the    World, ACM Transactions on Graphics, Vol. 31, No. 4 (2012) p 65 (8    pp.).

Other related writings include the following, each appended toapplication 61/872,494:

-   Abbas, et al, Hair Removal Methods: a Comparative Study for    Dermoscopy Images, Biomedical Signal Processing and Control 6.4,    2011, pp. 395-404;-   Armstrong et al, Crowdsourcing for Research Data Collection in    Rosacea, Dermatology Online Journal, Vol. 18, No. 3, March, 2012;-   Baeg et al, Organic Light Detectors—Photodiodes and Phototransistors    Advanced Materials, Volume 25, Issue 31, Aug. 21, 2013;-   Bellazzi, et al, Predictive Data Mining in Clinical Medicine—Current    Issues and Guidelines, Int'l J. of Medical Informatics, V. 77, 2008,    pp. 81-97;-   BioGames—A Platform for Crowd-Sourced Biomedical Image Analysis and    Telediagnosis, Games Health, Oct. 1, 2012, pp. 373-376;-   Cruz, et al, Applications of Machine Learning in Cancer Prediction    and Prognosis, Cancer Infom., No. 2, 2006, pp. 59-77;-   Csurka, et al, Visual Categorization with Bags of Keypoints, ECCV,    Workshop on Statistical Learning in Computer Vision, 2004;-   Dalal, et al, Histograms of Oriented Gradients for Human Detection,    IEEE Conference on Computer Vision and Pattern Recognition, pp.    886-893, 2005;-   di Leo, Automatic Diagnosis of Melanoma: A Software System Based on    the 7-Point Check-List, Proc. 43d Hawaii Int'l Conf. on System    Sciences, 2010;-   Foncubierta-Rodriguez et al, Ground Truth Generation in Medical    Imaging, Proc. of the ACM Multimedia 2012 workshop on Crowdsourcing    for Multimedia, pp. 9-14;-   Fuketa, et al, Large-Area and Flexible Sensors with Organic    Transistors, 5th IEEE Int'l Workshop on Advances in Sensors and    Interfaces, 2013;-   Jacobs et al, Focal Stack Compositing for Depth of Field Control,    Stanford Computer Graphics Laboratory Technical Report 2012-1;-   Johnson, et al, Retrographic Sensing for the Measurement of Surface    Texture and Shape, 2009 IEEE Conf. on Computer Vision and Pattern    Recognition;-   Kaliyadan, Teledermatology Update—Mobile Teledermatology, World    Journal of Dermatology, May 2, 2013, pp. 11-15;-   Liu, et al, Incorporating Clinical Metadata with Digital Image    Features for Automated Identification of Cutaneous Melanoma,    pre-print from British Journal of Dermatology, Jul. 31, 2013;-   Lyons, et al, Automatic classification of single facial images, IEEE    Trans. on Pattern Analysis and Machine Intelligence, Vol. 21, No.    12, 1999, pp. 1357-1362;-   Parsons, et al, Noninvasive Diagnostic Techniques for the Detection    of Skin Cancers, in Comparative Effectiveness Technical Briefs, No.    11, US Agency for Healthcare Research and Quality, September, 2011;-   Seal, et al, Automated Thermal Face Recognition Based on Minutiae    Extraction, Int. J. Computational Intelligence Studies, 2013, No. 2,    133-156;-   Vellido, et al, Neural Networks and Other Machine Learning Methods    in Cancer Research, in Computational and Ambient Intelligence,    Springer, 2007, pp. 964-971;-   Wadhawan, et al, SkinScan: A Portable Library for Melanoma Detection    on Handheld Devices, Proc. IEEE Int'l Symp. on Biomedical Imaging,    Mar. 30, 2011, pp. 133-136;-   Wolf et al, Diagnostic Inaccuracy of Smartphone Applications for    Melanoma Detection, JAMA Dermatology, Vol. 149, No. 4, April 2013;    and-   Zeng, et al, Colour and Tolerance of Preferred Skin Colours, Color    and Imaging Conference, Society for Imaging Science and Technology,    2010.

The artisan is presumed to be familiar with such art.

This specification details a variety of arrangements. It should beunderstood that the methods, elements and concepts detailed inconnection with one arrangement can be combined with the methods,elements and concepts detailed in connection with other embodiments.(For example, methods, principles and arrangements described inconnection with imagery can be applied in connection with audio, etc.,and vice versa. Similarly, polarized light can be used advantageously inembodiments employing SLAM or SFM techniques, and in detecting robustfeature points. Etc.) Likewise with features from the cited references.While some such arrangements have been particularly described, many havenot—due to the large number of permutations and combinations. However,implementation of all such combinations is straightforward to theartisan from the provided teachings.

While this disclosure has detailed particular ordering of acts andparticular combinations of elements, it will be recognized that othercontemplated methods may re-order acts (possibly omitting some andadding others), and other contemplated combinations may omit someelements and add others, etc.

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated (e.g., omitting variousfeatures of a complete system).

While certain aspects of the technology have been described by referenceto illustrative methods, it will be recognized that apparatusesconfigured to perform the acts of such methods are also contemplated aspart of applicant's inventive work. Likewise, other aspects have beendescribed by reference to illustrative apparatus, and the methodologyperformed by such apparatus is likewise within the scope of the presenttechnology. Still further, tangible computer readable media containinginstructions for configuring a processor or other programmable system toperform such methods is also expressly contemplated.

The present specification should be read in the context of the citedreferences. (The reader is presumed to be familiar with such priorwork.) Those references disclose technologies and teachings thatapplicant intends be incorporated into embodiments of the presenttechnology, and into which the technologies and teachings detailedherein be incorporated.

To provide a comprehensive disclosure, while complying with thestatutory requirement of conciseness, applicantincorporates-by-reference each of the documents referenced herein. (Suchmaterials are incorporated in their entireties, even if cited above inconnection with specific of their teachings. For example, while patentpublication 20110301441 was referenced in connection with purpose-builtimaging hardware, other technologies disclosed in that publication canalso be used advantageously herein.)

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only, and should not be taken aslimiting the scope of the technology. Rather, applicant claims all suchmodifications as may come within the scope and spirit of the attachedclaims and equivalents thereof.

1. A method comprising the acts: capturing a first image of skin at afirst time, using a camera of a smartphone; capturing a second image ofsaid skin at a second time later than the first time; identifying pluralfeatures in one of said images; locating said plural features in theother of said images; transforming one or more of said images toco-register said plural features therebetween; and after saidtransforming, presenting a time-based sequence of images, on a displayof the smartphone, to a viewer.
 2. The method of claim 1 that furtherincludes capturing a third image of said skin at a third time later thanthe second time, and including said third image in said presentedtime-based sequence of images.
 3. The method of claim 1 in which saidplural features comprise one or more nevi.
 4. The method of claim 1 inwhich said plural features comprise one or more hair follicles.
 5. Themethod of claim 1 in which said plural features comprise one or morewrinkles.
 6. The method of claim 1 in which said plural featurescomprise one or more pigmented regions.
 7. The method of claim 1 inwhich said plural features include one or more features from below anoutermost layer of skin.
 8. The method of claim 1 that includesdetermining an area of a lesion on said skin, and avoiding said area inidentifying said plural features.
 9. The method of claim 8 that includesflooding said area with a uniform color, so that features are notidentified therein.
 10. The method of claim 1 that includes processingsaid captured images to remove depictions of hair therein beforeidentifying and locating said plural features.
 11. The method of claim10 wherein said time-based sequence of images presented to the viewerincludes depictions of hair.
 12. The method of claim 1 in which saidtransforming further includes applying color correction to one or moreof said images.
 13. The method of claim 1 in which said presentingincludes presenting different images in said sequence in response to asliding-finger motion by the viewer on said display.
 14. The method ofclaim 1 in which said transforming includes a transformation other thanrotation and/or affine transformation, to account for curvature of asurface of the skin.
 15. The method of claim 1 that includes presentingone of said presented images for a first duration, and presentinganother of said presented images for a second duration different thanthe first duration, in accordance with spans of time between captures ofdifferent images.
 16. The method of claim 1 that includes overlayingmeasurement information on said presented sequence of images.
 17. Themethod of claim 1 in which said presenting includes overlaying one ofsaid images on another of said images, as a ghosted augmentation. 18.The method of claim 17 that further includes displaying a user interfacethat enables the viewer to select a particular image for saidoverlaying.
 19. The method of claim 1 that includes analyzing saidtime-based sequence of images by comparing associated data withinformation from reference time-based sequences, for diagnosticpurposes.
 20. The method of claim 1 in which at least one of said imagesincludes color data other than red, green or blue.
 21. A methodcomprising the acts: capturing a first image of skin at a first time;capturing a live image of said skin at a second time later than thefirst time; identifying plural features in one of said images; locatingsaid plural features in the other of said images; transforming one ofsaid images to co-register said plural features between the first andlive images; and presenting said first and live images, after saidtransforming, on a display to a viewer.
 22. The method of claim 21 inwhich said presenting includes presenting one of said images as aghosted overlay on the other of said images.
 23. The method of claim 21in which said plural features comprise features selected from the listconsisting of nevi, hair follicles, wrinkles, pores, pigmented regions,and skin textures.
 24. The method of claim 21 in which said pluralfeatures include one or more features from below an outermost layer ofskin.
 25. The method of claim 21 in which said transforming employs ahardware processor in a smartphone.
 26. The method of claim 21 thatincludes determining an area of a lesion on said skin, and avoiding saidarea in identifying said plural features.
 27. The method of claim 21that includes processing said captured images to remove depictions ofhair therein before identifying and locating said plural features. 28.The method of claim 27 wherein said presented images include depictionsof hair.
 29. The method of claim 21 in which said transforming includesa transformation other than rotation and/or affine transformation, toaccount for curvature of a surface of the skin.
 30. The method of claim21 that further includes: capturing another image of said skin at a timebetween the first and second times; locating said plural features insaid another image; transforming said another image to co-register saidanother image with the first and live images; and including saidtransformed another image in said presentation of images on the displayto the viewer.
 31. A non-transitory computer readable medium containingsoftware instructions for configuring one or more processors of asmartphone to perform acts including: capturing a first image of skin ata first time, using a camera of the smartphone; capturing a second imageof said skin at a second time later than the first time; identifyingplural features in one of said images; locating said plural features inthe other of said images; transforming one or more of said images toco-register said plural features therebetween; and after saidtransforming, presenting a time-based sequence of images, on a displayof the smartphone, to a viewer.